Rclone sync keeps recopying files because of precision loss in modification date?

Help and Support
1

What is the problem you are having with rclone?

rclone sync keeps recopying my files. Example:

❯ rclone sync -v -P --bwlimit 1m ~/SomeLocalDirectory 'someremote:someFolder'
…
2021/06/02 21:10:50 INFO  : Starting bandwidth limiter at 1MBytes/s
2021-06-02 21:10:50 INFO  : .localized: Copied (replaced existing)
2021-06-02 21:10:50 INFO  : .DS_Store: Copied (replaced existing)
2021-06-02 21:10:50 INFO  : SomeFile: Copied (replaced existing)
2021-06-02 21:10:50 INFO  : SomeFile2: Copied (replaced existing)
2021-06-02 21:10:50 INFO  : SomeFile3: Copied (replaced existing)
…

❯ rclone check ~/SomeLocalDirectory 'someremote:someFolder'

Enter configuration password:
password:
2021/06/02 21:13:46 NOTICE: Encrypted drive 'someremote:someFolder': 0 differences found
2021/06/02 21:13:46 NOTICE: Encrypted drive 'someremote:someFolder': 29 hashes could not be checked
2021/06/02 21:13:46 NOTICE: Encrypted drive 'someremote:someFolder': 29 matching files

❯ rclone sync -v -P --bwlimit 1m ~/SomeLocalDirectory 'someremote:someFolder’
…
2021/06/02 21:16:47 INFO  : Starting bandwidth limiter at 1MBytes/s
2021-06-02 21:16:48 INFO  : .localized: Copied (replaced existing)
2021-06-02 21:16:48 INFO  : .DS_Store: Copied (replaced existing)
2021-06-02 21:16:48 INFO  : SomeFile: Copied (replaced existing)
2021-06-02 21:16:48 INFO  : SomeFile2: Copied (replaced existing)
2021-06-02 21:16:48 INFO  : SomeFile3: Copied (replaced existing)

I think it's happening because there seems to be a slight loss of precision in the modification timestamps.

What is your rclone version (output from rclone version)

rclone v1.55.1

  • os/type: darwin
  • os/arch: amd64
  • go/version: go1.16.3
  • go/linking: dynamic
  • go/tags: cmount

Which OS you are using and how many bits (eg Windows 7, 64 bit)

macOS Big Sur (64-bit)

Which cloud storage system are you using? (eg Google Drive)

A NAS drive that is mounted to my local filesystem at /Volumes/SomeNAS/, which has a subdirectory that I wrapped in a crypt remote I called someremote in the example above.

The rclone config contents with secrets removed.

[someremote]
type = crypt
remote = /Volumes/SomeNAS/Vault
password = *** ENCRYPTED ***
password2 = *** ENCRYPTED ***

A log from the command with the -vv flag

2021/06/02 21:23:43 DEBUG : Using config file from "/Users/username/.config/rclone/rclone.conf"
2021/06/02 21:23:43 INFO  : Starting bandwidth limiter at 1MBytes/s
2021/06/02 21:23:43 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "sync" "-vv" "-P" "--bwlimit" "1m" "/path/to/SomeLocalDirectory" "someremote:someFolder"]
2021/06/02 21:23:43 DEBUG : Creating backend with remote "/path/to/SomeLocalDirectory"
2021/06/02 21:23:43 DEBUG : Creating backend with remote "someremote:someFolder"
2021/06/02 21:23:43 DEBUG : Creating backend with remote "/Volumes/SomeNAS/Vault/1ng4lhh37ctq29ora13evpcju8/cgs5cerki3l8c0uja902dlf4ms"
2021-06-02 21:23:43 DEBUG : .localized: Modification times differ by -905.913ms: 2021-04-27 19:48:39.905913 +0200 CEST, 2021-04-27 19:48:39 +0200 CEST
2021-06-02 21:23:43 DEBUG : .DS_Store: Modification times differ by -372.658481ms: 2021-06-02 21:04:22.372658481 +0200 CEST, 2021-06-02 21:04:22 +0200 CEST
2021-06-02 21:23:43 DEBUG : .DS_Store: MD5 = 4883226281cc694b5f48b51f7072ebb5 OK
2021-06-02 21:23:43 DEBUG : .localized: MD5 = 1b60c0209d40aa33b1c9b4a537344cca OK
2021-06-02 21:23:43 INFO  : .DS_Store: Copied (replaced existing)
2021-06-02 21:23:43 INFO  : .localized: Copied (replaced existing)
2021-06-02 21:23:43 DEBUG : SomeFileXYZ: Modification times differ by -896.443035ms: 2021-04-27 22:21:06.896443035 +0200 CEST, 2021-04-27 22:21:06 +0200 CEST
2021-06-02 21:23:43 DEBUG : SomeFileABC: Modification times differ by -206.961319ms: 2021-04-27 22:21:07.206961319 +0200 CEST, 2021-04-27 22:21:07 +0200 CEST

How do I prevent rclone sync from recopying files when their modification dates are off by less than a second due to what appears to be precision loss?

2

hello and welcome to the forum,

perhaps this can help
https://rclone.org/docs/#modify-window-time

--modify-window=1s

and best to test with --dry-run

3

Thank you! Very useful. Explicitly setting this solved my issue. Interestingly, the documentation says the following:

The default is 1ns unless this is overridden by a remote. For example OS X only stores modification times to the nearest second so if you are reading and writing to an OS X filing system this will be 1s by default.

I was using macOS. Any chance there's a difference between documentation and reality?

4

glad to help,

sure, that is possible.
for example, the reality is that i am a very good person but the documentation contradicts that :wink:
but seriously, perhaps the crypt overrides the osx?

5

Not sure how I can verify that's the case, but that would indeed be a logical explanation. Regardless, I was writing a backup script so I can easily add the flag there.

6

I'd imagine that was probably true before APFS and probably still is on older systems, but with new systems and APFS, it has a more granular time.

You can compare by running a rclone lsl on a file on an old MacOS file system and a new APFS one.

7

I checked: my local directories have millisecond precision while my SMB mount at /Volumes/SomeNAS/ has the milliseconds set to 0 on every file. Haven't tried mounting through AFP yet.

8

APFS is fine sorry as I should have mentioned I tested on that.

      213 2021-06-02 12:30:06.485039469 hosts
9

Oh, I was mixing up APFS and AFP. My local drive was indeed running the Apple File System.

10

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.