What is the problem you are having with rclone?
When I mount a sftp backend, then open any file for viewing (e.g. a document) and then perform a rclone sync between that backend and another backend, the file is considered as modified.
What is your rclone version (output from rclone version)
$ rclone version
rclone v1.55.1
- os/type: linux
- os/arch: amd64
- go/version: go1.16.3
- go/linking: static
- go/tags: none
Which OS you are using and how many bits (eg Windows 7, 64 bit)
Linux Mint 20.1 64-bit
Which cloud storage system are you using? (eg Google Drive)
Hetzner Storage Box via sftp
The command you were trying to run (eg rclone copy /tmp remote:tmp)
rclone --dry-run -vv --progress sync sftp-backend: b2-backend:
The remote is mounted via systemd:
[Unit]
Description=Hetzner Storage Box - encrypted union
AssertPathIsDirectory=/mnt/rclone/sftp-crypt
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
ExecStart=/usr/bin/rclone mount \
--config=/root/.config/rclone/rclone.conf \
--allow-other \
--vfs-cache-mode full \
--checksum \
sftp-crypt:/ /mnt/rclone/sftp-crypt
ExecStop=/root/scripts/rclone_unmount.sh /mnt/rclone/sftp-crypt
Restart=always
RestartSec=10
[Install]
WantedBy=default.target
The rclone config contents with secrets removed.
[sftp-backend-01]
type = sftp
host = <sftp-host>
user = <sftp-user>
key_file = <sftp-key>
port = <sftp-port>
md5sum_command = md5 -r
sha1sum_command = sha1 -r
[sftp-backend-02]
type = sftp
host = <sftp-host2>
user = <sftp-user2>
port = <sftp-port2>
key_file = <sftp-key2>
md5sum_command = md5 -r
sha1sum_command = sha1 -r
[sftp-union]
type = union
upstreams = sftp-backend-01:data sftp-backend-02:data
action_policy = all
create_policy = all
search_policy = all
[sftp-crypt]
type = crypt
remote = sftp-union:crypt
password = <pwd>
password2 = <salt>
A log from the command with the -vv flag
2021-06-03 11:41:06 DEBUG : aaaaaaaaaa.csv: Size and modification time the same (differ by 0s, within tolerance 1s)
2021-06-03 11:41:06 DEBUG : aaaaaaaaaa.csv: Unchanged skipping
2021-06-03 11:41:06 DEBUG : bbbbbbbbbb.pdf: Size and modification time the same (differ by 2ms, within tolerance 1s)
2021-06-03 11:41:06 DEBUG : bbbbbbbbbb.pdf: Unchanged skipping
2021-06-03 11:41:06 DEBUG : test.txt: Modification times differ by -669h20m32.351s: 2021-06-03 11:39:10 +0200 CEST, 2021-05-06 12:18:37.649 +0000 UTC
2021-06-03 11:41:06 DEBUG : cccccccccc.pdf: Size and modification time the same (differ by 377ms, within tolerance 1s)
2021-06-03 11:41:06 DEBUG : cccccccccc.pdf: Unchanged skipping
2021-06-03 11:41:06 DEBUG : dddddddddd.pdf: Size and modification time the same (differ by 0s, within tolerance 1s)
2021-06-03 11:41:06 NOTICE: test.txt: Skipped copy as --dry-run is set (size 38)
2021-06-03 11:41:06 DEBUG : dddddddddd.pdf: Unchanged skipping
The file test.txt had a timestamp of 2021-05-06 12:18:37.649 +0000 UTC, but after I opened it in an editor and then closed it again, without saving the file, the timestamp changed to the "current timestamp" (i.e. the time when I opened the file).
I also attached a screenshot of the filemanagers where you can clearly see that the files in both locations had the exact same timestamp right before I opened it.
This only occurs with sftp and AFAIK only with the Hetzner Storage boxes. When using a Backblaze (B2) backend or a Wasabi (S3) backend then the timestamps don't change when opening a file for reading.
I also tested with 2 local linux servers as an encrypted sftp union backend and until now I couldn't reproduce this same behaviour.
So to summarize: an rclone sync works fine, incl correct timestamps, but as soon as I start opening files on the Hetzner storage boxes backend the files will have a different timestamp and therefore will be synced (while they shouldn't because they didn't actually change at all)
