Rclone SFTP no hashes found

What is the problem you are having with rclone?

I'm setting up a NAS using my raspberry pi and open media vault. I'm trying to copy and sync some data from my mac book using SFTP. Connecting to remote and copying data is working, but hash comparison isn't. When using rclone check or rclone md5sum for example, it gives out an error.

I read here in the doc that SFTP supports both md5 and sha1.

I can use md5sum "locally" when i'm ssh-ed into my raspberry pi, as well as the equivalent command md5 on my mac.

i have a mac book pro 2019, running big sur, and a raspberry pi 4 with open media vault 5

What is your rclone version (output from rclone version)

I'm using rclone v1.56.2

Which cloud storage system are you using? (eg Google Drive)

I'm using my raspberry pi on my local network with open media vault and connecting through SFTP.

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone md5sum "spi:/Wired/archive/00 admin/portraits" -vv

The rclone config contents with secrets removed.

these are my remotes, i am using the spi remote to connect to the server since sftp should support hashing.

[pi]
type = ftp
host = 192.168.1.102
user = linn
port = 21
pass = //////
concurrency = 1

[spi]
type = sftp
host = 192.168.1.102
user = linn
port = 222
pass = //////
md5sum_command = md5sum
sha1sum_command = none

A log from the command with the -vv flag

2021/10/12 15:02:47 DEBUG : rclone: Version "v1.56.2" starting with parameters ["rclone" "md5sum" "spi:/Wired/archive/00 admin/portraits" "-vv"]
2021/10/12 15:02:47 DEBUG : Creating backend with remote "spi:/Wired/archive/00 admin/portraits"
2021/10/12 15:02:47 DEBUG : Using config file from "/Users/linn/.config/rclone/rclone.conf"
2021/10/12 15:02:47 DEBUG : sftp://linn@192.168.1.102:222//Wired/archive/00 admin/portraits: New connection 192.168.1.101:54024->192.168.1.102:222 to "SSH-2.0-OpenSSH_7.9p1 Raspbian-10+deb10u2+rpt1"
2021/10/12 15:02:47 DEBUG : sftp cmd = /Wired/archive/00\ admin/portraits/._IMG_2748.png
2021/10/12 15:02:47 DEBUG : ._IMG_2748.png: Failed to calculate md5 hash: Process exited with status 1 ()
                                  ._IMG_2748.png
2021/10/12 15:02:47 DEBUG : sftp cmd = /Wired/archive/00\ admin/portraits/._IMG_9904.JPG
2021/10/12 15:02:47 DEBUG : ._IMG_9904.JPG: Failed to calculate md5 hash: Process exited with status 1 ()
                                  ._IMG_9904.JPG
2021/10/12 15:02:47 DEBUG : sftp cmd = /Wired/archive/00\ admin/portraits/._self.png
2021/10/12 15:02:47 DEBUG : ._self.png: Failed to calculate md5 hash: Process exited with status 1 ()
                                  ._self.png
2021/10/12 15:02:47 DEBUG : sftp cmd = /Wired/archive/00\ admin/portraits/41.jpeg
2021/10/12 15:02:47 DEBUG : 41.jpeg: Failed to calculate md5 hash: Process exited with status 1 ()
                                  41.jpeg
2021/10/12 15:02:47 DEBUG : sftp://linn@192.168.1.102:222//Wired/archive/00 admin/portraits: New connection 192.168.1.101:54025->192.168.1.102:222 to "SSH-2.0-OpenSSH_7.9p1 Raspbian-10+deb10u2+rpt1"
2021/10/12 15:02:47 DEBUG : sftp cmd = /Wired/archive/00\ admin/portraits/IMG_2748.png
2021/10/12 15:02:47 DEBUG : IMG_2748.png: Failed to calculate md5 hash: Process exited with status 1 ()
                                  IMG_2748.png
2021/10/12 15:02:47 DEBUG : sftp cmd = /Wired/archive/00\ admin/portraits/IMG_9904.JPG
2021/10/12 15:02:47 DEBUG : IMG_9904.JPG: Failed to calculate md5 hash: Process exited with status 1 ()
                                  IMG_9904.JPG
2021/10/12 15:02:47 DEBUG : sftp cmd = /Wired/archive/00\ admin/portraits/self.png
2021/10/12 15:02:47 DEBUG : self.png: Failed to calculate md5 hash: Process exited with status 1 ()
                                  self.png
2021/10/12 15:02:47 DEBUG : sftp://linn@192.168.1.102:222//Wired/archive/00 admin/portraits: New connection 192.168.1.101:54026->192.168.1.102:222 to "SSH-2.0-OpenSSH_7.9p1 Raspbian-10+deb10u2+rpt1"
2021/10/12 15:02:47 DEBUG : sftp cmd = /Wired/archive/00\ admin/portraits/.DS_Store
2021/10/12 15:02:47 DEBUG : .DS_Store: Failed to calculate md5 hash: Process exited with status 1 ()
                                  .DS_Store
2021/10/12 15:02:47 DEBUG : sftp://linn@192.168.1.102:222//Wired/archive/00 admin/portraits: New connection 192.168.1.101:54027->192.168.1.102:222 to "SSH-2.0-OpenSSH_7.9p1 Raspbian-10+deb10u2+rpt1"
2021/10/12 15:02:47 DEBUG : sftp cmd = /Wired/archive/00\ admin/portraits/._41.jpeg
2021/10/12 15:02:47 DEBUG : ._41.jpeg: Failed to calculate md5 hash: Process exited with status 1 ()
                                  ._41.jpeg
2021/10/12 15:02:48 DEBUG : sftp cmd = /Wired/archive/00\ admin/portraits/._.DS_Store
2021/10/12 15:02:48 DEBUG : ._.DS_Store: Failed to calculate md5 hash: Process exited with status 1 ()
                                  ._.DS_Store
2021/10/12 15:02:48 DEBUG : 40 go routines active

Thank you for your time!

hello and welcome to the forum,

not sure what the exact problem is.

want to let you know that rclone itself can act as a sftp server
https://rclone.org/commands/rclone_serve_sftp/

You want to put in the path to md5sum so it knows where to find it.

felix@gemini:~$ which md5sum
/usr/bin/md5sum

or

etexter@seraphite ~ % which md5
/sbin/md5

On Linux and a Mac as examples.

hi,

hope you do not mind, just curious about that, as i am working on my linux skills.

if rclone cannot find the md5sum file, rclone would output
Failed to calculate md5 hash: Process exited with status 127 (bash: line 1: /usr/bin/md5sumz: No such file or directory)

whereas in this case, seems like rclone found and tried to execute the md5sum file.
Process exited with status 1 ()

It depends on how things work on a PI as I don't have one.

Step 1) validate the md5sum command is properly in the path and using right one
Step 2) log into PI and validate the command and see if it runs

Could be a restricted shell or many things going on so I like to start at the basics and make sure we are using the right binary.

Hi, thanks for the reply.

I added the path to config like you said

md5sum_command = /usr/bin/md5sum

which is where it is located on the pi, the destination computer. it hasn't solved the problem yet.

can you explain more what you mean by validating that the command is properly in the path? since the pi finds the command and can execute it when i'm logged into the pi, do you mean that i need to make sure that rclone somehow finds this command?

On the PI, run:

where md5sum

Validate the path matches.

If that's correct, run a test on a file you moved over:

felix@gemini:/tmp$ which md5sum
/usr/bin/md5sum
felix@gemini:/tmp$ /usr/bin/md5sum /tmp/hosts
f900f4098877238fbdb59e6910286672  /tmp/hosts

If that works, re-run the rclone command with a single file and add -vv and share the full output.

pi@raspberrypi:/sftp/linn/Wired/archive/00 admin $ which md5sum
/usr/bin/md5sum
pi@raspberrypi:/sftp/linn/Wired/archive/00 admin $ /usr/bin/md5sum CV.pdf
e2f91ef6341aff2926ecf1bc7fda0663  CV.pdf

yeah I tried doing that and works locally on my may as well as on the pi...

md5sum_command = /usr/bin/md5sum

the rclone config also seems to be correct

this is my result trying md5 with rclone remotely

linn@macbook-las ~ % rclone md5sum "spi:/Wired/archive/00 admin/portraits/41.jpeg" -vv
2021/10/14 00:55:22 DEBUG : rclone: Version "v1.56.2" starting with parameters ["rclone" "md5sum" "spi:/Wired/archive/00 admin/portraits/41.jpeg" "-vv"]
2021/10/14 00:55:22 DEBUG : Creating backend with remote "spi:/Wired/archive/00 admin/portraits/41.jpeg"
2021/10/14 00:55:22 DEBUG : Using config file from "/Users/linn/.config/rclone/rclone.conf"
2021/10/14 00:55:22 DEBUG : sftp://linn@192.168.1.102:222//Wired/archive/00 admin/portraits/41.jpeg: New connection 192.168.1.101:59967->192.168.1.102:222 to "SSH-2.0-OpenSSH_7.9p1 Raspbian-10+deb10u2+rpt1"
2021/10/14 00:55:23 DEBUG : fs cache: adding new entry for parent of "spi:/Wired/archive/00 admin/portraits/41.jpeg", "spi:/Wired/archive/00 admin/portraits"
2021/10/14 00:55:23 DEBUG : .DS_Store: Excluded
2021/10/14 00:55:23 DEBUG : IMG_9904.JPG: Excluded
2021/10/14 00:55:23 DEBUG : self.png: Excluded
2021/10/14 00:55:23 DEBUG : ._IMG_2748.png: Excluded
2021/10/14 00:55:23 DEBUG : ._41.jpeg: Excluded
2021/10/14 00:55:23 DEBUG : IMG_2748.png: Excluded
2021/10/14 00:55:23 DEBUG : ._IMG_9904.JPG: Excluded
2021/10/14 00:55:23 DEBUG : ._self.png: Excluded
2021/10/14 00:55:23 DEBUG : ._.DS_Store: Excluded
2021/10/14 00:55:23 DEBUG : sftp cmd = /Wired/archive/00\ admin/portraits/41.jpeg
2021/10/14 00:55:23 DEBUG : 41.jpeg: Failed to calculate md5 hash: Process exited with status 1 ()
                                  41.jpeg
2021/10/14 00:55:23 DEBUG : 13 go routines active

there is another post with the same issue
https://forum.rclone.org/t/strict-sftp-environment-makes-it-impossible-to-do-checksum/26939

have you modified sshd_config or using ForceCommand internal-sftp

What this is saying is that the md5sum process failed somehow - not sure exactly why.

However a common problem is that the SFTP root isn't the same as the SSH root

That is what the path_override setting is for - check out the docs here to see if they apply to you

https://rclone.org/sftp/#sftp-path-override