OneDrive sync checking very slow

Help and Support
10

tl;dr, this should be a one-time issue, let rclone do its thing, as explained below.
tho, as mentioned, a workaround is --modify-window 1s

this is do to rclone having to calculate the hash for each local file.
a consequence of changes onedrive made to the handling of modtimes.

 ./rclone_v1.65.1 backend features onedrive: | grep "Precision"
        "Precision": 1000000000,
./rclone_v1.68.2 backend features onedrive: | grep "Precision"
        "Precision": 1000000,

rclone uses Precision, in nanoseconds, as value for --modify-window

let's use a real file from the debug log you posted,

# check modtimes, notice modtimes do not match
DEBUG : GameData/duplicati-20240211T092112Z.dlist.zip.aes: Modification times differ by -681.7093ms: 2024-02-11 02:21:18.6817093 -0700 MST, 2024-02-11 09:21:18 +0000 UTC
# check hash of the files
DEBUG : GameData/duplicati-20240211T092112Z.dlist.zip.aes: quickxor = 00eddfe3594014a059648cdb10be041f71b4e5b9 OK
# update the modtime
INFO  : GameData/duplicati-20240211T092112Z.dlist.zip.aes: Updated modification time in destination
# double check the files are equivalent
DEBUG : GameData/duplicati-20240211T092112Z.dlist.zip.aes: Unchanged skipping

681ms is 681000000ns

* v1.65.1 - 681000000ns is less    than 	1000000000ns
* v1.68.2 - 681000000ns is greater than 	1000000ns

in both versions, the modtime has changed, the files are not identical. should rclone re-copy the file or not?

* v1.65.1 - rclone considers the files are equivalent, rclone does nothing.
* v1.68.2 - rclone considers the files not equivalent, rclone has to do something.

so, let's do something
rclone is cloud based, can be expensive to re-copy a file, uses internet bandwidth, api calls, unnecessary new version of files, etc..

  • the dumb way, re-upload the file.
  • the smart way, notice the file sizes are identical. then compare the hashes.
    if the hashes match, then no need to slow, expensive re-upload the file, just quick, cheap change of modtime.

the downside to being smart is rclone has to calculate the hash of each local file.
that takes time and slow down the entire sync command.
but hopefully, this is a one-time painful upgrade from an old version of rclone to the new version of rclone.

3 Likes