What is the problem you are having with rclone?
Hi at all,
whenever I want to transmit a file from the rclone host (SFTP) to a remote NAS (Synology) I encounter this error:
failed to run "md5sum /volume1/dms_backup/test1.txt": Permission denied
Run the command 'rclone version' and share the full output of the command.
rclone --version
rclone v1.61.0
- os/version: debian 11.5 (64 bit)
- os/kernel: 5.15.60-2-pve (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.19.4
- go/linking: static
- go/tags: none
Which cloud storage system are you using? (eg Google Drive)
SFTP on Synology NAS
The command you were trying to run (eg rclone copy /tmp remote:tmp
)
sudo rclone copy -vv --retries 1 --config /root/.config/rclone/rclone.conf /mnt/paperless_ngx/backup/test1.txt nas_dms:dms_backup --sftp-path-override /volume1/dms_backup
I also tried variations of the command to ensure it has nothing to do with the path values (e.g. because the docs state to use a leading slash for the remote when the remote is a Synology):
# With leading slash for remote path
sudo rclone copy -vv --retries 1 --config /root/.config/rclone/rclone.conf /mnt/paperless_ngx/backup/test1.txt nas_dms:/dms_backup --sftp-path-override /volume1/dms_backup
# Without path override
sudo rclone copy -vv --retries 1 --config /root/.config/rclone/rclone.conf /mnt/paperless_ngx/backup/test1.txt nas_dms:dms_backup
sudo rclone copy -vv --retries 1 --config /root/.config/rclone/rclone.conf /mnt/paperless_ngx/backup/test1.txt nas_dms:/dms_backup
On the NAS itself I tried to manually execute the md5sum command to ensure there is no problem with the user in question:
sudo -H -u paperless bash -c 'md5sum /volume1/dms_backup/somefile'
c7d226536fee164edd75392207f6b89d /volume1/dms_backup/somefile
The rclone config contents with secrets removed.
[nas_dms]
type = sftp
host = 192.168.2.102
user = paperless
pass = ...
shell_type = unix
md5sum_command = md5sum
sha1sum_command = sha1sum
A log from the command with the -vv
flag
$ sudo rclone copy -vv --retries 1 --config /root/.config/rclone/rclone.conf /mnt/paperless_ngx/backup/test1.txt nas_dms:dms_backup --sftp-path-override /volume1/dms_backup
2023/03/18 16:02:29 DEBUG : rclone: Version "v1.61.0" starting with parameters ["rclone" "copy" "-vv" "--retries" "1" "--config" "/root/.config/rclone/rclone.conf" "/mnt/paperless_ngx/backup/test1.txt" "nas_dms:dms_backup" "--sftp-path-override" "/volume1/dms_backup"]2023/03/18 16:02:29 DEBUG : Creating backend with remote "/mnt/paperless_ngx/backup/test1.txt"
2023/03/18 16:02:29 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2023/03/18 16:02:29 DEBUG : fs cache: adding new entry for parent of "/mnt/paperless_ngx/backup/test1.txt", "/mnt/paperless_ngx/backup"
2023/03/18 16:02:29 DEBUG : Creating backend with remote "nas_dms:dms_backup"
2023/03/18 16:02:29 DEBUG : nas_dms: detected overridden config - adding "{GUxpu}" suffix to name
2023/03/18 16:02:29 DEBUG : sftp://paperless@192.168.2.102:22/dms_backup: New connection 192.168.2.31:60524->192.168.2.102:22 to "SSH-2.0-OpenSSH_8.2"
2023/03/18 16:02:29 DEBUG : sftp://paperless@192.168.2.102:22/dms_backup: Shell type "unix" from config
2023/03/18 16:02:29 DEBUG : sftp://paperless@192.168.2.102:22/dms_backup: Relative path resolved to "/dms_backup"
2023/03/18 16:02:29 DEBUG : sftp://paperless@192.168.2.102:22/dms_backup: Using root directory "/dms_backup"
2023/03/18 16:02:29 DEBUG : fs cache: renaming cache item "nas_dms:dms_backup" to be canonical "nas_dms{GUxpu}:dms_backup"
2023/03/18 16:02:29 DEBUG : test1.txt: Need to transfer - File not found at Destination
2023/03/18 16:02:29 DEBUG : sftp://paperless@192.168.2.102:22/dms_backup: Shell path redirected to "/volume1/dms_backup/test1.txt" with option path_override
2023/03/18 16:02:29 DEBUG : sftp://paperless@192.168.2.102:22/dms_backup: Running remote command: md5sum /volume1/dms_backup/test1.txt
2023/03/18 16:02:29 ERROR : test1.txt: Failed to calculate dst hash: failed to calculate md5 hash: failed to run "md5sum /volume1/dms_backup/test1.txt": Permission denied, please try again.: Process exited with status 1
2023/03/18 16:02:29 ERROR : test1.txt: corrupted on transfer: md5 hash differ "d41d8cd98f00b204e9800998ecf8427e" vs ""
2023/03/18 16:02:29 INFO : test1.txt: Removing failed copy
2023/03/18 16:02:29 ERROR : Attempt 1/1 failed with 2 errors and: corrupted on transfer: md5 hash differ "d41d8cd98f00b204e9800998ecf8427e" vs ""
2023/03/18 16:02:29 INFO :
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Errors: 2 (retrying may help)
Elapsed time: 0.3s
2023/03/18 16:02:29 DEBUG : 13 go routines active
2023/03/18 16:02:29 DEBUG : sftp://paperless@192.168.2.102:22/dms_backup: Closing 1 unused connections
2023/03/18 16:02:29 Failed to copy with 2 errors: last error was: corrupted on transfer: md5 hash differ "d41d8cd98f00b204e9800998ecf8427e" vs ""
asdffdsa
(jojothehumanmonkey)
March 18, 2023, 3:45pm
2
hi,
in this post, i show how to get rclone to calculate the md5 on a synbox.
https://forum.rclone.org/t/on-some-sftp-servers-e-g-synology-the-paths-are-different-for-ssh-and-sftp-so-the-hashes-cant-be-calculated-properly/36506
if i intentionally set --sftp-path-override
to the wrong path, i get the same error as you.
not an expert with linux but
another possible issue, that you run sudo
to get md5sum
to work.
i would think user paperless
, needs permission to access the files without root.
Hi, thanks for supporting. I'll check this. Concerning the path: Maybe this is due to that I am using a shared folder. Maybe this is a symlink or whatever under the hood. Would that matter? I'll check that.
asdffdsa
(jojothehumanmonkey)
March 18, 2023, 5:02pm
4
shared folder should not make a difference.
this works on a shared folder.
rclone copy file.ext nas_dms:dms_backup --sftp-path-override=/volume1/dms_backup -vv
2023/03/18 13:00:58 DEBUG : Setting --config "C:\\data\\rclone\\rclone.conf" from environment variable RCLONE_CONFIG="C:\\data\\rclone\\rclone.conf"
2023/03/18 13:00:58 DEBUG : rclone: Version "v1.61.1" starting with parameters ["C:\\data\\rclone\\rclone.exe" "copy" "C:\\data\\rclone\\file.ext" "nas_dms:dms_backup" "--sftp-path-override=/volume1/dms_backup" "-vv"]
2023/03/18 13:00:58 DEBUG : Creating backend with remote "C:\\data\\rclone\\file.ext"
2023/03/18 13:00:58 DEBUG : Using config file from "C:\\data\\rclone\\rclone.conf"
2023/03/18 13:00:58 DEBUG : fs cache: adding new entry for parent of "C:\\data\\rclone\\file.ext", "//?/C:/data/rclone"
2023/03/18 13:00:58 DEBUG : Creating backend with remote "nas_dms:dms_backup"
2023/03/18 13:00:58 DEBUG : nas_dms: detected overridden config - adding "{GUxpu}" suffix to name
2023/03/18 13:00:58 DEBUG : sftp://user99@100.115.37.34:22/dms_backup: New connection 100.109.123.23:49400->100.115.37.34:22 to "SSH-2.0-OpenSSH_8.2"
2023/03/18 13:00:58 DEBUG : sftp://user99@100.115.37.34:22/dms_backup: Shell type "unix" from config
2023/03/18 13:00:58 DEBUG : sftp://user99@100.115.37.34:22/dms_backup: Relative path resolved to "/dms_backup"
2023/03/18 13:00:58 DEBUG : sftp://user99@100.115.37.34:22/dms_backup: Using root directory "/dms_backup"
2023/03/18 13:00:58 DEBUG : fs cache: renaming cache item "nas_dms:dms_backup" to be canonical "nas_dms{GUxpu}:dms_backup"
2023/03/18 13:00:58 DEBUG : file.ext: Need to transfer - File not found at Destination
2023/03/18 13:00:58 DEBUG : sftp://user99@100.115.37.34:22/dms_backup: Shell path redirected to "/volume1/dms_backup/file.ext" with option path_override
2023/03/18 13:00:58 DEBUG : sftp://user99@100.115.37.34:22/dms_backup: Running remote command: md5sum /volume1/dms_backup/file.ext
2023/03/18 13:00:58 DEBUG : sftp://user99@100.115.37.34:22/dms_backup: Remote command result: c039965eb7fcc80be96c2a578bf8bace /volume1/dms_backup/file.ext
2023/03/18 13:00:58 DEBUG : file.ext: Parsed hash: c039965eb7fcc80be96c2a578bf8bace
2023/03/18 13:00:58 DEBUG : file.ext: md5 = c039965eb7fcc80be96c2a578bf8bace OK
2023/03/18 13:00:58 INFO : file.ext: Copied (new)
2023/03/18 13:00:58 INFO :
Transferred: 16.940 KiB / 16.940 KiB, 100%, 0 B/s, ETA -
Transferred: 1 / 1, 100%
Elapsed time: 0.4s
2023/03/18 13:00:58 DEBUG : 12 go routines active
2023/03/18 13:00:58 DEBUG : sftp://user99@100.115.37.34:22/dms_backup: Closing 1 unused connections
One more thing: Can you tell which DSM version it is that you are running?
One more thing after the one thing: Do you know which exact command is run by rclone to get the hash? I mean the complete one like ssh user@host ... 'md5sum file'?
Tried to read it from the GitHub repo but had no luck so far.
Thanks!!!
asdffdsa
(jojothehumanmonkey)
March 18, 2023, 5:36pm
6
Thanks again.
I think I figured it out. But I don't think I like it...
First things first: It is not a problem of the user not being able to run the md5sum command:
paperless@nas3:/$ md5sum /volume1/dms_backup/somefile
c7d226536fee164edd75392207f6b89d /volume1/dms_backup/somefile
BUT: To be able to get to this point I recognized that my user paperless had no shell configured.
cat /etc/passwd
[...]
paperless:x:1027:100::/var/services/homes/paperless:/sbin/nologin
[...]
Changing that to
paperless:x:1027:100::/var/services/homes/paperless:/bin/sh
made the rclone command succeed.
But that implies one of the following two things:
@asdffdsa : Is your user within the admin group? admin users are the only group that are allowed to get SSH access. So they might also be the only ones that get a shell. I saw no other option within the Synology config that looked like it would enable a shell for non admin users Or did you somewhen enable the shell for your user? Otherwise I really cannot imagine how your user got a shell...
asdffdsa
(jojothehumanmonkey)
March 18, 2023, 8:01pm
8
yes
good question, i would post that to the synology forum.
I checked a few posts out there and they all state that SSH access is only granted to admin users and that way the shell /bin/sh is only set for them not for unpriv'ed users. Soooo thats maybe worth a mention in the docs that next to the plain SFTP permissions a real shell is neccessary (at least that this is a thing on synology NASes)?
asdffdsa
(jojothehumanmonkey)
March 18, 2023, 8:13pm
10
perhaps use AllowUsers
in /etc/ssh/sshd_config
From what I read that might not be ideal too. This config file might be overridden (like other system files are) whenever the Synology boots up. And I read about default permissions of user data that are set to 777 on file system level so a shell would mean access to all files... Did not validate that. But either way that would be a not supported workaround. And I learned the hard way to use a Synology only the intended way So I guess the more stable option would be to ... Turn off hash checking or to use a completely different approach... again... Maybe SMB/CIFS... Since FTP(S) in conjunction with my Synology is also out of scope I am slowly running out of options here
asdffdsa
(jojothehumanmonkey)
March 18, 2023, 8:31pm
12
stev-io:
might be overridden
yes, i read that also, but might be worth a try
stev-io:
Turn off hash checking
never!
agreed!
rclone serve sftp
asdffdsa:
clone serve sftp
Thanks for the hint. Does that basically mean to run rclone serve sftp on the Synology NAS?
asdffdsa
(jojothehumanmonkey)
March 18, 2023, 9:45pm
14
correct, the rclone sftp server will caclulate the hash without using a shell.
on the client, with remote nas_dms:
, set shell_type = none