I am having an issue in the latest stable version of rclone that is causing md5sum checks to fail on my Synology NAS running the latest Synology software versions.
The error seems to indicate that the destination path being passed to the md5sum command is being double escaped so a whitespace in Cold Storage is becoming Cold\\ Storage instead of Cold\ Storage and when I run the failing command via ssh manually it tries to run 2 separate commands with the second being just the rest of the path Storage\... and fails. The md5sum command fails first since Cold is not a file or directory.
I am using valid sftp path overrides and everything else looks in order except the logged command used for md5sum.
I tested version 1.55.1 and it doesn't have this issue so it may be a regression of sorts? I also tested 1.58.1 because of the PR that added support for windows shells for sftp and that also did not work with the same errors even though I think v1.58.1 is right before that PR was landed.
Run the command 'rclone version' and share the full output of the command.
rclone v1.68.1
os/version: darwin 15.1 (64 bit)
os/kernel: 24.1.0 (arm64)
os/type: darwin
os/arch: arm64 (ARMv8 compatible)
go/version: go1.23.1
go/linking: dynamic
go/tags: none
Which cloud storage system are you using? (eg Google Drive)
SFTP
The command you were trying to run (eg rclone copy /tmp remote:tmp)
This is an artifact of how the error message is being logged I think. I suspect it is being logged as %q which will put it in quotes and escape backslashes.
I doubt very much there is an issue with escaping in general as I'm sure someone would have noticed before. It could be to do with the path override which is a little used feature.
I suspect some interaction between the remote shell and rclone is confusing things.
2024/10/28 11:27:10 DEBUG : rclone: Version "v1.68.1" starting with parameters ["rclone" "--config=/Users/blainemiller/.config/rclone/rclone.conf" "--retries=1" "-vv" "copy" "test/" "millerhome:Documents/test/" "--sftp-path-override" "/volume1/Documents"]
2024/10/28 11:27:10 DEBUG : Creating backend with remote "test/"
2024/10/28 11:27:10 DEBUG : Using config file from "/Users/blainemiller/.config/rclone/rclone.conf"
2024/10/28 11:27:10 DEBUG : fs cache: renaming cache item "test/" to be canonical "/Users/blainemiller/Downloads/test"
2024/10/28 11:27:10 DEBUG : Creating backend with remote "millerhome:Documents/test/"
2024/10/28 11:27:10 DEBUG : millerhome: detected overridden config - adding "{eCeme}" suffix to name
2024/10/28 11:27:10 DEBUG : sftp://...@...:.../Documents/test/: New connection ...:54105->...:... to "SSH-2.0-OpenSSH_8.2"
2024/10/28 11:27:11 DEBUG : sftp://...@...:.../Documents/test/: Shell type "unix" from config
2024/10/28 11:27:11 DEBUG : sftp://blaine@millerhome.hawk-bowfin.ts.net:722/Documents/test/: Relative path resolved to "/Documents/test"
2024/10/28 11:27:11 DEBUG : sftp://...@...:.../Documents/test/: Using root directory "/Documents/test"
2024/10/28 11:27:11 DEBUG : fs cache: renaming cache item "millerhome:Documents/test/" to be canonical "millerhome{eCeme}:Documents/test/"
2024/10/28 11:27:11 DEBUG : Added delayed dir = "Cold Storage", newDst=<nil>
2024/10/28 11:27:11 DEBUG : Added delayed dir = "Cold Storage/foo", newDst=<nil>
2024/10/28 11:27:11 DEBUG : Cold Storage/foo/test.txt: Need to transfer - File not found at Destination
2024/10/28 11:27:11 DEBUG : sftp://...@...:.../Documents/test/: Waiting for checks to finish
2024/10/28 11:27:11 DEBUG : sftp://blaine@millerhome.hawk-bowfin.ts.net:722/Documents/test/: Waiting for transfers to finish
2024/10/28 11:27:11 DEBUG : sftp://...@...:.../Documents/test/: Shell path redirected to "/volume1/Documents/Cold Storage/foo/test.txt.506b3609.partial" with option path_override
2024/10/28 11:27:11 DEBUG : sftp://...@...:.../Documents/test/: Running remote command: /bin/md5sum /volume1/Documents/Cold\ Storage/foo/test.txt.506b3609.partial
2024/10/28 11:27:11 ERROR : Cold Storage/foo/test.txt.506b3609.partial: Failed to calculate dst hash: failed to calculate md5 hash: failed to run "/bin/md5sum /volume1/Documents/Cold\\ Storage/foo/test.txt.506b3609.partial": /bin/md5sum: '/volume1/Documents/Cold Storage/foo/test.txt.506b3609.partial': No such file or directory: Process exited with status 1
2024/10/28 11:27:11 ERROR : Cold Storage/foo/test.txt.506b3609.partial: corrupted on transfer: md5 hashes differ src(Local file system at /Users/blainemiller/Downloads/test) "ab8bdec2fcdb279b31424eb7ec2699d1" vs dst(sftp://...@...:.../Documents/test/) ""
2024/10/28 11:27:11 INFO : Cold Storage/foo/test.txt.506b3609.partial: Removing failed copy
2024/10/28 11:27:11 INFO : Cold Storage/foo: Set directory modification time (using DirSetModTime)
2024/10/28 11:27:11 INFO : Cold Storage: Set directory modification time (using DirSetModTime)
2024/10/28 11:27:11 ERROR : Attempt 1/1 failed with 2 errors and: corrupted on transfer: md5 hashes differ src(Local file system at /Users/blainemiller/Downloads/test) "ab8bdec2fcdb279b31424eb7ec2699d1" vs dst(sftp://...@...:.../Documents/test/) ""
2024/10/28 11:27:11 INFO :
Transferred: 13 B / 13 B, 100%, 0 B/s, ETA -
Errors: 2 (retrying may help)
Elapsed time: 0.5s
2024/10/28 11:27:11 DEBUG : 17 go routines active
2024/10/28 11:27:11 DEBUG : sftp://...@...:.../Documents/test/: Closing 1 unused connections
2024/10/28 11:27:11 NOTICE: Failed to copy with 2 errors: last error was: corrupted on transfer: md5 hashes differ src(Local file system at /Users/blainemiller/Downloads/test) "ab8bdec2fcdb279b31424eb7ec2699d1" vs dst(sftp://...@...:.../Documents/test/) ""
I then tried varying a few different things with the --sftp-path-override and using the full path to the destination passed to rclone all the way to the end /volume1/Documents/iCloud Drive in my case seems to eliminates the issue.
not sure if its a bug or user error at this point but thanks for pointing me to a solution.
When connecting to an SFTP server, the absolute path might not always be resolved correctly by default. Using =@/volume1/ explicitly directs the SFTP client to treat /volume1/ as the absolute path, preventing potential issues with path resolution, especially in environments like Synology NAS or chrooted environments where SFTP users may have limited access or a restricted root directory.
The =@ syntax is essentially an SFTP path override that ensures the path you specify will be interpreted correctly by the SFTP server.
Without =@
Using /volume1/ alone might result in path errors or unexpected behavior if the SFTP client interprets it relative to the user's home directory or defaults to a different directory structure.
This syntax is particularly useful with tools like rclone when working with remote servers that require absolute path specifications for correct file handling.
I think it is user error with an incorrect setting on the —sftp-path-override flag and not a bug with rclone. Using the full path /volume1/Documents/iCloud\ Drive no longer has issues. I will try and test using the ‘@‘ option that was described to see if that helps as well. Thank you for helping me find a solution. I appreciate the explanation of the ‘@‘ as I thought I read the docs about that parameter and don’t remember that @ being mentioned there. It’s probably in the docs somewhere though.
Just confirmed —sftp-path-override works using the =@/volume1/ without needing the full path to the rclone destination of =/volume1/Documents/iCloud\ Drive/
That’s a nice thing to learn. Thank you!
I might try and set that via a config option instead of needing to set the flag each time.
Totally worked added:
path_override = @/volume1/
to rclone.conf and I no longer need to set a flag for it.