Rclone sync does not properly preserve modification times when syncing to ecryptfs mounts

Suspected Bug
1

What is the problem you are having with rclone?

When using sync to a path on an ecryptfs mount, rclone does not preserve the milliseconds portion of the file timestamp. This results in rclone continually thinking that files need their modification time resynced.

Ecryptfs is capable of millisecond-resolution mod timestamps, as validated by touch:

$ touch foo
$ stat foo
...
Access: 2021-08-08 21:37:41.668700846 -0800
Modify: 2021-08-08 21:37:41.668700846 -0800
Change: 2021-08-08 21:37:41.668700846 -0800

What is your rclone version (output from rclone version)

Tested on v1.56.0 and v1.57.0-beta.5611.c32d5dd1f.

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Ubuntu Bionic

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

$ rclone sync -i gdrive:/200802_002.doc ./gdrive

After syncing the file, I can confirm the mtimes do not match:

$ rclone lsl ./gdrive/200802_002.doc
    36864 2014-03-18 13:11:36.000000000 200802_002.doc
$ rclone lsl gdrive:/200802_002.doc
    36864 2014-03-18 13:11:36.343000000 200802_002.doc

The rclone config contents with secrets removed.

{
    "gdrive": {
        "client_id": "redacted",
        "client_secret": "redacted",
        "scope": "drive",
        "skip_gdocs": "true",
        "team_drive": "",
        "token": "redacted",
        "type": "drive"
    }
}

A log from the command with the -vv flag

The following log will show up every time I run the sync, despite syncing the modtime every time.

2021/08/08 21:36:12 DEBUG : 200802_002.doc: Modification times differ by -343ms: 2014-03-18 21:11:36.343 +0000 UTC, 2014-03-18 13:11:36 -0800 AKDT
2

Hi alienth,

Welcome to the forum, good first post!

You may be able to further narrow down the issue using these commands in an empty folder:

touch foo
rclone touch bar
rclone lsl .
ls -l .
3

Thanks! Command output is below. Looks like rclone touch also isn't doing the right thing - resulting file has only seconds resolution. All of the following was done in an ecryptfs-mounted directory:

$ touch foo

$ stat foo
  File: foo
  Size: 0         	Blocks: 16         IO Block: 4096   regular empty file
Device: 3eh/62d	Inode: 1835692     Links: 1
Access: (0664/-rw-rw-r--)  Uid: ( 1000/ alienth)   Gid: ( 1000/ alienth)
Access: 2021-08-08 22:59:50.881291603 -0800
Modify: 2021-08-08 22:59:50.881291603 -0800
Change: 2021-08-08 22:59:50.881291603 -0800
 Birth: -

$ rclone touch bar

$ rclone lsl .
        0 2021-08-08 23:00:06.000000000 bar
        0 2021-08-08 22:59:50.881291603 foo

$ ls -l

total 16
-rw-rw-r-- 1 alienth alienth 0 Aug  8 23:00 bar
-rw-rw-r-- 1 alienth alienth 0 Aug  8 22:59 foo

$ stat foo bar
  File: foo
  Size: 0         	Blocks: 16         IO Block: 4096   regular empty file
Device: 3eh/62d	Inode: 1835692     Links: 1
Access: (0664/-rw-rw-r--)  Uid: ( 1000/ alienth)   Gid: ( 1000/ alienth)
Access: 2021-08-08 22:59:50.881291603 -0800
Modify: 2021-08-08 22:59:50.881291603 -0800
Change: 2021-08-08 22:59:50.881291603 -0800
 Birth: -
  File: bar
  Size: 0         	Blocks: 16         IO Block: 4096   regular empty file
Device: 3eh/62d	Inode: 1837002     Links: 1
Access: (0664/-rw-rw-r--)  Uid: ( 1000/ alienth)   Gid: ( 1000/ alienth)
Access: 2021-08-08 23:00:06.000000000 -0800
Modify: 2021-08-08 23:00:06.000000000 -0800
Change: 2021-08-08 23:00:06.569623328 -0800
 Birth: -
4

I suspected so, what if you try the commands directly in the local file system (without encryptfs)?

5

Yep, rclone touch works AOK and maintains the milliseconds on touched files directly on normal filesystems. I also tested more nuanced scenarios like overlay and full timestamp resolution is maintained. So far I've only been able to reproduce this on ecryptfs.

Reviewing the syscalls, looks like rclone and touch both use the utimensat syscall. However, rclone sets the time explicitly, whereas touch passes NULL:

syscall from rclone:

utimensat(AT_FDCWD, "/home/alienth/tmp/bar", [{tv_sec=1628501018, tv_nsec=492082171} /* 2021-08-09T01:23:38.492082171-0800 */, {tv_sec=1628501018, tv_nsec=492082171} /* 2021-08-09T01:23:38.492082171-0800 */], 0) = 0

syscall from touch (fd 0 is our bar file):

utimensat(0, NULL, NULL, 0)             = 0

Docs from utimensat:

If times is NULL, then both timestamps are set to the current time.

Perhaps this is a bug with ecryptfs only when using utimensat with an explicit timestamp?

End result of all of this of course is that rclone continually wants to synchronize timestamps when using sync against a path from an ecryptfs mount.

6

Hah, it is an ecryptfs bug. You'll find the bug report on launchpad at bug #1890486 (sorry, cannot link).

utimensat truncates timestamp to whole second granularity

7

Interesting further diving: I tested with rsync and the resulting files it synchronizes to an ecryptfs mount also don't maintain millisecond-level resolution. However, surprisingly a subsequent rsync doesn't try to re-sync the mtime, even with the --times flag.

Turns out this is because rsync has a default modify-window of 0, which translates to integer seconds:

--modify-window=NUM, -@
When comparing two timestamps, rsync treats the timestamps as being equal if they differ by no more than the modify-window value. The default is 0, which matches just integer seconds.

As such, rsync doesn't happen to get bitten by this bad ecryptfs behaviour (when using default flags), while rclone does.

8

Great analysis!

rclone also has a --modify-window.

I haven't tried it, but it seems to have similar functionality.

9

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.