Rclone updates files that were only accessed during sync

Suspected Bug
1

Hey!

My setup is the following: I have an encrypted volume (historically it’s CryFS, but could change if needed) that I am syncing to S3 with Rclone. So what I do in my script is:

#Mount entrypted volume
cryfs /mnt/media/encrypted/volume /home/mine/decrypted
#Rclone it to S3
rclone -v sync /home/mine/decrypted amazonS3:

It basically does it’s job, but the problem is it updates not only modified and new files on remote, but also files that I merely accessed on the local volume. I.e. if I open the file read-only and then sync, rclone will say

2017/05/01 02:30:15 INFO  : (some file): Copied (replaced existing)

although ll of that file will say

-rw-rw-r-- 1 user group    491520 Apr 12 21:17 (some file)

Apr 12 being before the previous sync.

So the question is - how rclone decides whether to update the file or not and are there any possible hiccups while working with FUSE local volumes? I understand that this problem can be more of CryFS, but at least I’d like to understand the exact cause. Some configs exist maybe that command how rclone diffs files?

Thanks a lot!

2

rclone should be using the modify time of the file.

The mtime shouldn’t change if you access the file, eg

$ stat x
  File: 'x'
  Size: 207       	Blocks: 8          IO Block: 4096   regular file
Device: fd01h/64769d	Inode: 6042726     Links: 1
Access: (0664/-rw-rw-r--)  Uid: (  502/     ncw)   Gid: (  502/     ncw)
Access: 2017-04-26 11:27:41.401602293 +0100
Modify: 2017-04-26 11:26:35.017434935 +0100
Change: 2017-04-26 11:26:35.017434935 +0100
 Birth: -
$ cat x > /dev/null
$ stat x
  File: 'x'
  Size: 207       	Blocks: 8          IO Block: 4096   regular file
Device: fd01h/64769d	Inode: 6042726     Links: 1
Access: (0664/-rw-rw-r--)  Uid: (  502/     ncw)   Gid: (  502/     ncw)
Access: 2017-05-02 14:24:34.906232279 +0100
Modify: 2017-04-26 11:26:35.017434935 +0100
Change: 2017-04-26 11:26:35.017434935 +0100
 Birth: -

Can you do a similar test on cryfs?

3

Nick, thanks a lot! The issue was not Rclone, nor CryFS, but the SFTP client I used (which is FAR with Netbox plugin).

Long story short, every time I download a file from a mounted partition with this client, this happened:

user@Serv:~/mount/1/decrypted$ stat file.pdf
  File: 'file.pdf'
  Size: 57400           Blocks: 0          IO Block: 16384  regular file
Device: 2ch/44d Inode: 66          Links: 1
Access: (0664/-rw-rw-r--)  Uid: ( 1000/user)   Gid: ( 1000/user)
Access: 2017-05-03 12:43:05.434725378 +0300
Modify: 2017-04-05 23:22:52.814915665 +0300
Change: 2017-04-05 23:22:52.814915690 +0300
 Birth: -

(download file through SFTP)

user@Serv:~/mount/1/decrypted$ stat file.pdf
  File: 'file.pdf'
  Size: 98304           Blocks: 0          IO Block: 16384  regular file
Device: 2ch/44d Inode: 66          Links: 1
Access: (0664/-rw-rw-r--)  Uid: ( 1000/user)   Gid: ( 1000user)
Access: 2017-05-03 12:44:53.048720634 +0300
Modify: 2017-04-05 23:22:52.814915665 +0300
Change: 2017-04-05 23:22:52.814915690 +0300
 Birth: -

Note the change of filesize between the two. So of course Rclone synced it at next run.
Filesize does not change if the same client downloads another file from outside a mounted partition (i.e. /home/user/dir), so has to be something to do with SFTP and FUSE maybe. The workaround for me was to switch to SCP instead - which works fine.

In any case, thanks for your advice - got me digging :wink:

4

Glad you got to the bottom of it!

I’ll just note rclone can talk sftp too if that is helpful!