Md5sum on Windows can't handle any unusual characters, breaking copy, sync, etc

What is the problem you are having with rclone?

When transferring files to Windows hosts via SFTP, md5sum fails to run for any file with an unusual character in its filename. This prevents syncing from working and pollutes the remote with .partial files.

"Unusual" characters includes things like the apostrophe ’, registered trademark ®, or any emoji :gift:. These are characters that are representible in UTF-8 and which other tools do transfer without a problem, e.g. robocopy.

When md5sum fails to run, rclone declares it a corrupted transfer, leaves behind a partial file and sees an empty string for the hash:

2024/06/22 14:30:12 ERROR : /Colors.bin.hijoqij4.partial: corrupted on transfer: md5 hashes differ src(Local file system at //?/c:/home) "63daacc93458669390062bf97e71c529" vs dst(sftp://me@desk:8314//c:/home/) ""

Run the command 'rclone version' and share the full output of the command.

rclone v1.67.0

  • os/version: Microsoft Windows 11 Pro 23H2 (64 bit)
  • os/kernel: 10.0.22631.3737 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.22.4
  • go/linking: static
  • go/tags: cmount

Welcome to the forum.

The checksum handling in SFTP backend is a bit special, as explained here.

Your configuration is highly relevant to be able to understand what happens, especially the shell_type and md5sum_command parameters. Run rclone config redacted (remove any remaining secrets) and post the output. Also posting a debug log could give crucial information.

rclone copy d:\zork win_sftp_local:/c:/home -vv --inplace 
DEBUG : rclone: Version "v1.67.0" starting with parameters ["c:\\data\\rclone\\rclone.exe" "copy" "d:\\zork" "win_sftp_local:/c:/home" "-vv" "--inplace"]
DEBUG : Creating backend with remote "d:\\zork"
DEBUG : Using config file from "c:\\data\\rclone\\rclone.conf"
DEBUG : fs cache: renaming cache item "d:\\zork" to be canonical "//?/d:/zork"
DEBUG : Creating backend with remote "win_sftp_local:/c:/home"
DEBUG : sftp://user01@127.0.0.1:9999//c:/home: New connection 127.0.0.1:2084->127.0.0.1:9999 to "SSH-2.0-OpenSSH_for_Windows_8.6"
DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Shell type "cmd" from config
DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Using root directory "/c:/home"
DEBUG : '®🎁: Need to transfer - File not found at Destination
DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Waiting for checks to finish
DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Waiting for transfers to finish
DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Shell path adjusted to "c:/home/'®🎁" (set option path_override to override)
DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Running remote command: c:/data/rclone/rclone md5sum "c:/home/'®🎁"
DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Remote command result: d41d8cd98f00b204e9800998ecf8427e  '®🎁
DEBUG : '®🎁: Parsed hash: d41d8cd98f00b204e9800998ecf8427e
DEBUG : '®🎁: md5 = d41d8cd98f00b204e9800998ecf8427e OK
INFO  : '®🎁: Copied (new)

Great test!

The question is which md5sum command is used by the OP. I'm wondering if maybe the server is configured with PowerShell as shell, and remote config is shell_type=powershell, and no checksum command is found (md5sum, md5, rclone md5sum) so that it resorts to the built-in PowerShell script block - maybe that does not handle these characters?

"The praise of the praiseworthy"
Neil Peart - October 2006 issue of Modern Drummer

well, not sure that is the issue, as the error occurs when the source dir is empty, There was nothing to transfer

so i went with what worked, forever trusted dumb dos.

[win_sftp_local]
type = sftp
host = 127.0.0.1
port = 9999
user = user01
shell_type = cmd
md5sum_command = //?/c:/data/rclone/rclone md5sum
key_file = //?/c:/data/c/combined/.self/ts-ubunut01/keys/id_ed25519
sha1sum_command = none

in the forum, you are the powershell guy

[win_sftp_local_powershell]
type = sftp
host = 127.0.0.1
port = 9999
user = user01
shell_type = powershell
key_file = //?/c:/data/c/combined/.self/ts-ubunut01/keys/id_ed25519

and then ran this command. note: the source dir is empty, nothing to transfer

rclone copy //?/d:/zork win_sftp_local_powershell:/c:/home -vv --inplace  
2024/06/22 17:33:55 DEBUG : rclone: Version "v1.67.0" starting with parameters ["c:\\data\\rclone\\rclone.exe" "copy" "//?/d:/zork" "win_sftp_local_powershell:/c:/home" "-vv" "--inplace"]
2024/06/22 17:33:55 DEBUG : Creating backend with remote "//?/d:/zork"
2024/06/22 17:33:55 DEBUG : Using config file from "c:\\data\\rclone\\rclone.conf"
2024/06/22 17:33:55 DEBUG : Creating backend with remote "win_sftp_local_powershell:/c:/home"
2024/06/22 17:33:55 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: New connection 127.0.0.1:3621->127.0.0.1:9999 to "SSH-2.0-OpenSSH_for_Windows_8.6"
2024/06/22 17:33:55 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Shell type "powershell" from config
2024/06/22 17:33:55 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Using root directory "/c:/home"
2024/06/22 17:33:55 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Checking default md5 hash commands
2024/06/22 17:33:55 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Running remote command: md5sum
2024/06/22 17:33:55 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Hash command skipped: failed to run "md5sum": 'md5sum' is not recognized as an internal or external command,
operable program or batch file.: Process exited with status 1
2024/06/22 17:33:55 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Running remote command: md5 -r
2024/06/22 17:33:55 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Hash command skipped: failed to run "md5 -r": 'md5' is not recognized as an internal or external command,
operable program or batch file.: Process exited with status 1
2024/06/22 17:33:55 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Running remote command: rclone md5sum
2024/06/22 17:33:55 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Hash command skipped: failed to run "rclone md5sum": 'rclone' is not recognized as an internal or external command,
operable program or batch file.: Process exited with status 1
2024/06/22 17:33:55 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Running remote command: Get-FileHash -Algorithm MD5 -InputStream ([System.IO.MemoryStream]::new()) -ErrorAction Stop|Select-Object -First 1 -ExpandProperty Hash|ForEach-Object{$_.ToLower()}
2024/06/22 17:33:55 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Hash command skipped: failed to run "Get-FileHash -Algorithm MD5 -InputStream ([System.IO.MemoryStream]::new()) -ErrorAction Stop|Select-Object -First 1 -ExpandProperty Hash|ForEach-Object{$_.ToLower()}": 'Get-FileHash' is not recognized as an internal or external command,
operable program or batch file.: Process exited with status 255
2024/06/22 17:33:55 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Checking default sha1 hash commands
2024/06/22 17:33:55 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Running remote command: sha1sum
2024/06/22 17:33:55 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Hash command skipped: failed to run "sha1sum": 'sha1sum' is not recognized as an internal or external command,
operable program or batch file.: Process exited with status 1
2024/06/22 17:33:55 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Running remote command: sha1 -r
2024/06/22 17:33:56 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Hash command skipped: failed to run "sha1 -r": 'sha1' is not recognized as an internal or external command,
operable program or batch file.: Process exited with status 1
2024/06/22 17:33:56 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Running remote command: rclone sha1sum
2024/06/22 17:33:56 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Hash command skipped: failed to run "rclone sha1sum": 'rclone' is not recognized as an internal or external command,
operable program or batch file.: Process exited with status 1
2024/06/22 17:33:56 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Running remote command: Get-FileHash -Algorithm SHA1 -InputStream ([System.IO.MemoryStream]::new()) -ErrorAction Stop|Select-Object -First 1 -ExpandProperty Hash|ForEach-Object{$_.ToLower()}
2024/06/22 17:33:56 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Hash command skipped: failed to run "Get-FileHash -Algorithm SHA1 -InputStream ([System.IO.MemoryStream]::new()) -ErrorAction Stop|Select-Object -First 1 -ExpandProperty Hash|ForEach-Object{$_.ToLower()}": 'Get-FileHash' is not recognized as an internal or external command,
operable program or batch file.: Process exited with status 255
2024/06/22 17:33:56 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Setting hash command for md5 to "none" (set sha1sum_command to override)
2024/06/22 17:33:56 DEBUG : Saving config "md5sum_command" in section "win_sftp_local_powershell" of the config file
2024/06/22 17:33:56 DEBUG : Keeping previous permissions for config file: -rw-rw-rw-
2024/06/22 17:33:56 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Setting hash command for sha1 to "none" (set md5sum_command to override)
2024/06/22 17:33:56 DEBUG : Saving config "sha1sum_command" in section "win_sftp_local_powershell" of the config file
2024/06/22 17:33:56 DEBUG : Keeping previous permissions for config file: -rw-rw-rw-
2024/06/22 17:33:56 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Waiting for checks to finish
2024/06/22 17:33:56 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Waiting for transfers to finish
2024/06/22 17:33:56 INFO  : There was nothing to transfer

Server is in this test not configured to use PowerShell, I think, which is what I had in mind:

nice, that worked

rclone copy //?/d:/zork win_sftp_local_powershell:/c:/home -vv --inplace  --retries=1 --low-level-retries=1 
2024/06/22 19:02:21 DEBUG : rclone: Version "v1.67.0" starting with parameters ["c:\\data\\rclone\\rclone.exe" "copy" "//?/d:/zork" "win_sftp_local_powershell:/c:/home" "-vv" "--inplace" "--retries=1" "--low-level-retries=1"]
2024/06/22 19:02:21 DEBUG : Creating backend with remote "//?/d:/zork"
2024/06/22 19:02:21 DEBUG : Using config file from "c:\\data\\rclone\\rclone.conf"
2024/06/22 19:02:21 DEBUG : Creating backend with remote "win_sftp_local_powershell:/c:/home"
2024/06/22 19:02:21 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: New connection 127.0.0.1:4531->127.0.0.1:9999 to "SSH-2.0-OpenSSH_for_Windows_8.6"
2024/06/22 19:02:21 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Shell type "powershell" from config
2024/06/22 19:02:21 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Using root directory "/c:/home"
2024/06/22 19:02:21 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Checking default md5 hash commands
2024/06/22 19:02:21 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Running remote command: md5sum
2024/06/22 19:02:22 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Hash command skipped: failed to run "md5sum": md5sum : The term 'md5sum' is not recognized as the name of a cmdlet, function, script file, or 
operable program. Check the spelling of the name, or if a path was included, verify that the path 
is correct and try again.
At line:1 char:1
+ md5sum
+ ~~~~~~
    + CategoryInfo          : ObjectNotFound: (md5sum:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException: Process exited with status 1
2024/06/22 19:02:22 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Running remote command: md5 -r
2024/06/22 19:02:22 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Hash command skipped: failed to run "md5 -r": md5 : The term 'md5' is not recognized as the name of a cmdlet, function, script file, or 
operable program. Check the spelling of the name, or if a path was included, verify that the path 
is correct and try again.
At line:1 char:1
+ md5 -r
+ ~~~
    + CategoryInfo          : ObjectNotFound: (md5:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException: Process exited with status 1
2024/06/22 19:02:22 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Running remote command: rclone md5sum
2024/06/22 19:02:23 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Hash command skipped: failed to run "rclone md5sum": rclone : The term 'rclone' is not recognized as the name of a cmdlet, function, script file, or 
operable program. Check the spelling of the name, or if a path was included, verify that the path 
is correct and try again.
At line:1 char:1
+ rclone md5sum
+ ~~~~~~
    + CategoryInfo          : ObjectNotFound: (rclone:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException: Process exited with status 1
2024/06/22 19:02:23 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Running remote command: Get-FileHash -Algorithm MD5 -InputStream ([System.IO.MemoryStream]::new()) -ErrorAction Stop|Select-Object -First 1 -ExpandProperty Hash|ForEach-Object{$_.ToLower()}
2024/06/22 19:02:24 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Remote command result: d41d8cd98f00b204e9800998ecf8427e
2024/06/22 19:02:24 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Hash command accepted
2024/06/22 19:02:24 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Checking default sha1 hash commands
2024/06/22 19:02:24 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Running remote command: sha1sum
2024/06/22 19:02:24 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Hash command skipped: failed to run "sha1sum": sha1sum : The term 'sha1sum' is not recognized as the name of a cmdlet, function, script file, or 
operable program. Check the spelling of the name, or if a path was included, verify that the path 
is correct and try again.
At line:1 char:1
+ sha1sum
+ ~~~~~~~
    + CategoryInfo          : ObjectNotFound: (sha1sum:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException: Process exited with status 1
2024/06/22 19:02:24 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Running remote command: sha1 -r
2024/06/22 19:02:25 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Hash command skipped: failed to run "sha1 -r": sha1 : The term 'sha1' is not recognized as the name of a cmdlet, function, script file, or 
operable program. Check the spelling of the name, or if a path was included, verify that the path 
is correct and try again.
At line:1 char:1
+ sha1 -r
+ ~~~~
    + CategoryInfo          : ObjectNotFound: (sha1:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException: Process exited with status 1
2024/06/22 19:02:25 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Running remote command: rclone sha1sum
2024/06/22 19:02:25 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Hash command skipped: failed to run "rclone sha1sum": rclone : The term 'rclone' is not recognized as the name of a cmdlet, function, script file, or 
operable program. Check the spelling of the name, or if a path was included, verify that the path 
is correct and try again.
At line:1 char:1
+ rclone sha1sum
+ ~~~~~~
    + CategoryInfo          : ObjectNotFound: (rclone:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException: Process exited with status 1
2024/06/22 19:02:25 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Running remote command: Get-FileHash -Algorithm SHA1 -InputStream ([System.IO.MemoryStream]::new()) -ErrorAction Stop|Select-Object -First 1 -ExpandProperty Hash|ForEach-Object{$_.ToLower()}
2024/06/22 19:02:26 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Remote command result: da39a3ee5e6b4b0d3255bfef95601890afd80709
2024/06/22 19:02:26 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Hash command accepted
2024/06/22 19:02:26 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Setting hash command for md5 to "&{param($Path);Get-FileHash -Algorithm MD5 -LiteralPath $Path -ErrorAction Stop|Select-Object -First 1 -ExpandProperty Hash|ForEach-Object{\"$($_.ToLower())  ${Path}\"}}" (set sha1sum_command to override)
2024/06/22 19:02:26 DEBUG : Saving config "md5sum_command" in section "win_sftp_local_powershell" of the config file
2024/06/22 19:02:26 DEBUG : Keeping previous permissions for config file: -rw-rw-rw-
2024/06/22 19:02:26 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Setting hash command for sha1 to "&{param($Path);Get-FileHash -Algorithm SHA1 -LiteralPath $Path -ErrorAction Stop|Select-Object -First 1 -ExpandProperty Hash|ForEach-Object{\"$($_.ToLower())  ${Path}\"}}" (set md5sum_command to override)
2024/06/22 19:02:26 DEBUG : Saving config "sha1sum_command" in section "win_sftp_local_powershell" of the config file
2024/06/22 19:02:26 DEBUG : Keeping previous permissions for config file: -rw-rw-rw-
2024/06/22 19:02:26 DEBUG : '®🎁: Need to transfer - File not found at Destination
2024/06/22 19:02:26 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Waiting for checks to finish
2024/06/22 19:02:26 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Waiting for transfers to finish
2024/06/22 19:02:26 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Shell path adjusted to "c:/home/'®🎁" (set option path_override to override)
2024/06/22 19:02:26 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Running remote command: &{param($Path);Get-FileHash -Algorithm MD5 -LiteralPath $Path -ErrorAction Stop|Select-Object -First 1 -ExpandProperty Hash|ForEach-Object{"$($_.ToLower())  ${Path}"}} 'c:/home/''®🎁'
2024/06/22 19:02:26 DEBUG : sftp://user01@127.0.0.1:9999//c:/home: Remote command result: d41d8cd98f00b204e9800998ecf8427e  c:/home/'r??
2024/06/22 19:02:26 DEBUG : '®🎁: Parsed hash: d41d8cd98f00b204e9800998ecf8427e
2024/06/22 19:02:26 DEBUG : '®🎁: md5 = d41d8cd98f00b204e9800998ecf8427e OK
2024/06/22 19:02:26 INFO  : '®🎁: Copied (new)
2024/06/22 19:02:26 INFO  : 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         5.5s

and here is the resultant remote config

[win_sftp_local_powershell]
type = sftp
host = 127.0.0.1
port = 9999
user = user01
shell_type = powershell
key_file = //?/c:/data/c/combined/.self/ts-ubunut01/keys/id_ed25519
md5sum_command = &{param($Path);Get-FileHash -Algorithm MD5 -LiteralPath $Path -ErrorAction Stop|Select-Object -First 1 -ExpandProperty Hash|ForEach-Object{"$($_.ToLower())  ${Path}"}}
sha1sum_command = &{param($Path);Get-FileHash -Algorithm SHA1 -LiteralPath $Path -ErrorAction Stop|Select-Object -First 1 -ExpandProperty Hash|ForEach-Object{"$($_.ToLower())  ${Path}"}}

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.