#### What is the purpose of this change?
Windows 10, Server 2019, and later c…an run a SSH server that is a port of OpenSSH. See official [installation guide](https://docs.microsoft.com/en-us/windows-server/administration/openssh/openssh_install_firstuse).
Existing SFTP backend in rclone assumes there is a Unix shell, if any, on the server when executing remote commands. Windows SSH server *can* run with bash shell (Cygwin? WSL?), but defaults to CMD, and even more natural is probably to configure it to use PowerShell. Since both CMD and PowerShell uses different quoting/escaping rules from bash, and also different from each other, the existing remote command execution does not (necessarily) work. If staying away from checksums and the about command, which are the two places remote commands are used currently, the existing version should work. But this PR intends to fix also the missing pieces, such that Windows OpenSSH server can be said to be fully supported.
What it does:
- Introduces new config option `shell_type` with possible values `none|cmd|powershell|unix`
- Auto-detects shell when option `shell_type` is not explicitely set (config file, command-line, env-var)
- On first access it will execute a simple shell command and try to figure out if there is a "powershell" or a "cmd" behind the remote, and if neither of them are detected then assumes "unix".
- If a remote ssh session could not be created it sets value "none", which disables shell command execution.
- If remote ssh session creation succeeded, but the execution of the detection command failed (nonzero exit code, or any other errors), then assumes "unix".
- The result is stored back permanently to the config file, to avoid having to run the same detection command next time.
- The existing checksum command options was already doing this sort of thing.
- If explicitely specifying remote shell option, e.g. command line option `--sftp-shell-type=cmd`, then this will be used; the auto-detect logic will be skipped if this is first run, or else any previously stored config will be ignored.
- This option will *not* be saved back to the config file (which is also similar to existing handling of checksum command options).
- When shell is powershell, this PR also added default commands for checksum and about by using built-in PowerShell functionality, so these features will be supported by default without having to configure external commands.
- It works, but is quite verbose and looks a bit weird in the config file:
`md5sum_command = &{param($Path);Get-FileHash -Algorithm MD5 -LiteralPath $Path -ErrorAction Stop|Select-Object -First 1 -ExpandProperty Hash|ForEach-Object{"$($_.ToLower()) ${Path}"}}`
`sha1sum_command = &{param($Path);Get-FileHash -Algorithm SHA1 -LiteralPath $Path -ErrorAction Stop|Select-Object -First 1 -ExpandProperty Hash|ForEach-Object{"$($_.ToLower()) ${Path}"}}`
In addition:
- Added use of vendor-specific VFS statistics extension for about if available. Seems OpenSSH implements this on every platform, at least in the Windows port which I am working on here. This means the usage data can be retrieved directly with api function instead of running shell commands at all. Still falling back to existing shell command if necessary.
- ~Fixes an issue with session handling in hash calculation function. It would release the connection to connection pool premature, before using it to execute the hash command.~ (removed after same change was added to master in 099eff8891d1d7bd081bb1c9218ff5c0863921b2)
Beta builds:
- https://beta.rclone.org/branch/sftp-windows-server/
Notes:
- The autodetection logic is not bullet proof, but the idea is that it works in most cases and makes it convenient for users, and one can always override it from config in the few cases it fails.
Implementation details:
- Shell autodetection:
- It executes `echo ${ShellId}%ComSpec%`, which seems to be a single command to cover most cases.
- The expected output is:
- CMD: `${ShellId}C:\WINDOWS\system32\cmd.exe`
- PowerShell: `Microsoft.PowerShell%ComSpec%`
- Bash: `%ComSpec%`
- Based on this we can deduce:
- If output has prefix Microsoft.PowerShell I'm quite confident its PowerShell.
- If output does not have suffix %ComSpec% I'm pretty sure its cmd. Only cmd has the %variable% syntax, as far as I know.
- Could probably in theory have a cmd without ComSpec environment variable set, and then it will not be detected and fallback to Unix, but I think this is rare - and then user can always force a specific shell type in config file or command-line argument.
- No explicit check for Unix shell, this is the fallback if the cmd or powershell is not detected.
- If connecting to an "rclone serve sftp" instance it will always return `${ShellId}%ComSpec%`, and fall-back to unix shell, but this is correct as rclone supports "unix shell api" in this mode (it responds to df, md5sum, etc, and unescapes unix shell escaping performed by sftp remote).
- If configuring OpenSSH on Windows to use `ssh-shellhost.exe`, the autodetection logic fails, and user must force "cmd".
- The echo command returns with exit code 255, and on stdout: "ssh-shellhost cannot run 'echo ${ShellId}%ComSpec%', error: 2"
- Shell path conversion:
- If remote shell is powershell or cmd, then server is probably Windows. The sftp package converts everything to POSIX paths: Forward slashes, and absolute paths starts with a slash. An absolute path on a Windows server will then look like this "/C:/Windows/System32". We must remove the "/" prefix to make this a valid for shell commands. In case of PowerShell there is a possibility that it is a Unix server, with PowerShell Core shell, but assuming root folders with names such as "C:" are rare, we just take this risk, option path_override can always be used to work around corner cases.
- When implementing shell path conversion, a possible issue with the about command and override_path option was also fixed.
- Hashsum commands joined override_path with path to the file object relative to fs root.
- E.g. with `rclone sha1sum remote:/Path/To/File.txt --sftp-path-override=/a/b/c` the executed command is :
```
sha1sum /a/b/c/File.txt
```
- About command joined override_path with root.
- E.g. with `rclone about remote:/Path/To/File.txt --sftp-path-override=/a/b/c` the executed command would be wrong:
```
df -k /a/b/c/Path/To/File.txt
```
- Normally about command is run without path, e.g. `rclone about remote:`, and in these cases the result would be correct:
```
df -k /a/b/c
```
TODO:
- Name of the option: `shell`, `shell_type`, `remote_shell`, `remote_shell_type`,...? Initially used `remote_shell`, currently `shell_type`... I can't make up my mind which to use, so suggestions are welcome.
- If OpenSSH is configured to use ssh-shellhost.exe the shell autodetection fails.
- This is an included OpenSSH ShellHost, seems to be a VT100 adapter over Windows Console used for compatibility before support for this was added to the built-in console in Windows 10.
- When I tested this on a Win10, the autodetect logic failed so it ended up being set to "unix", but for paths to be escaped correctly with an md5sum_command etc it must be forced to "cmd". Will see if there is something to do, but if not I think/hope we can just leave this as a "known imperfection" (as my project manager likes to call it), as this seems to be a legacy shell setup anyway.
- Test with bash shell on Windows? E.g. Cygwin, MinGW or WSL..
- Evaluate logic with saving back autodetected value to config file.
- With the built-in powershell sum commands, there is a dependency between `shell_type` and `md5sum_command`/`sha1sum_command`. If running with `--sftp-shell-type=powershell` first time, then `md5sum_command` and `sha1sum_command` will get automatic values which are powershell commands and be saved to config file. But `shell_type` is not saved to config file. So if next time running with `--sftp-shell-type=cmd`, or without this option and it is autodetected to something different than powershell, then the `md5sum_command` and `sha1sum_command` that will be picked up from config file is not valid.
- But perhaps such cases can be considered user error?
OLD:
- Is "built-in" support for about-command with cmd shell possible?
- There is `fsutil volume diskfree c:`, but it requires admin privileges...?
- Could re-use the same as powershell, just execute it explicitely via powershell.exe.
- Less important after support for VFS statistics extension feature was introuced, this works regardless of shell - for servers that has this extension.
- String encoding
- UTF-8
- Everything should be UTF-8, Windows OpenSSH server seems to do conversion as needed.
- When executing remote shell command for checksum and path contains non-ascii characters, will it always be correct then?
- Have done *some* testing with special characters, haven't seen any issues so far - but there may be a ton of variables in play here..
- The sftp backend does not seem to use rclone's internal encoding, with Unicode replacements, and functions FromStandardName/ToStandardName...?
#### Was the change discussed in an issue or in the forum before?
Fixes #5758
#### Checklist
- [X] I have read the [contribution guidelines](https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#submitting-a-new-feature-or-bug-fix).
- [ ] I have added tests for all changes in this PR if appropriate.
- [X] I have added documentation for the changes if appropriate.
- [X] All commit messages are in [house style](https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#commit-messages).
- [X] I'm done, this Pull Request is ready for review :-)
