Using update and use-server-modtime when syncing causes a few files to be constantly reuploaded

Help and Support
1

What is the problem you are having with rclone?

I am syncing 50,000 or so files with a few calls to rsync to encrypted folders on iDrive e2. Everything runs very smoothly except that in one call, with 24K files, the same 20 files are constantly being reuploaded when I sync with update (-u) and use-server-modtime. When I call refresh-times the same 20 files in their own subdirectory have their times updated. The other 23,900+ are skipped as they should be whether or not I call refresh-times. None of my other buckets have this problem either.

I've also tried moving the files out of the source, re-syncing (which deleted the files), and then moving them back. The files still get reuploaded/updated.

Only if I drop use-server-modtime will the files be skipped, but that makes the sync process longer.

I have Verizon Fios and 192GB of RAM, which is why I have been liberal with my checkers, transfers, and buffers.

Run the command 'rclone version' and share the full output of the command.

rclone v1.69.2

  • os/version: Microsoft Windows 11 Pro 24H2 24H2 (64 bit)
  • os/kernel: 10.0.26100.4061 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.24.2
  • go/linking: static
  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

iDrive e2

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync "D:\Users\Parents\Pictures\Family Pictures" fampics: -vu --fast-list --use-server-modtime --fix-case --checkers 256 --transfers 32 --drive-chunk-size 256M --max-backlog=-1 --buffer-size 2G

The result is the following, no matter what I tried above.

c:\rclone>rclone sync "D:\Users\Parents\Pictures\Family Pictures" fampics: -vu --fast-list --use-server-modtime --fix-case --checkers 256 --transfers 32 --drive-chunk-size 256M --max-backlog=-1 --buffer-size 2G
2025/06/06 14:53:40 INFO  : Extended/Adler/Aryeh & Michal Bensinger/2009/Yutta Roiza Bensinger/Yutta Roiza Bensinger - 2009 - 01.JPG: Copied (replaced existing)
2025/06/06 14:53:40 INFO  : Extended/Adler/Aryeh & Michal Bensinger/2009/Yutta Roiza Bensinger/Yutta Roiza Bensinger - 2009 - 14.JPG: Copied (replaced existing)
2025/06/06 14:53:40 INFO  : Extended/Adler/Aryeh & Michal Bensinger/2009/Yutta Roiza Bensinger/Yutta Roiza Bensinger - 2009 - 08.JPG: Copied (replaced existing)
2025/06/06 14:53:40 INFO  : Extended/Adler/Aryeh & Michal Bensinger/2009/Yutta Roiza Bensinger/Yutta Roiza Bensinger - 2009 - 02.JPG: Copied (replaced existing)
2025/06/06 14:53:40 INFO  : Extended/Adler/Aryeh & Michal Bensinger/2009/Yutta Roiza Bensinger/Yutta Roiza Bensinger - 2009 - 20.JPG: Copied (replaced existing)
2025/06/06 14:53:40 INFO  : Extended/Adler/Aryeh & Michal Bensinger/2009/Yutta Roiza Bensinger/Yutta Roiza Bensinger - 2009 - 17.JPG: Copied (replaced existing)
2025/06/06 14:53:40 INFO  : Extended/Adler/Aryeh & Michal Bensinger/2009/Yutta Roiza Bensinger/Yutta Roiza Bensinger - 2009 - 16.JPG: Copied (replaced existing)
2025/06/06 14:53:40 INFO  : Extended/Adler/Aryeh & Michal Bensinger/2009/Yutta Roiza Bensinger/Yutta Roiza Bensinger - 2009 - 18.JPG: Copied (replaced existing)
2025/06/06 14:53:40 INFO  : Extended/Adler/Aryeh & Michal Bensinger/2009/Yutta Roiza Bensinger/Yutta Roiza Bensinger - 2009 - 04.JPG: Copied (replaced existing)
2025/06/06 14:53:40 INFO  : Extended/Adler/Aryeh & Michal Bensinger/2009/Yutta Roiza Bensinger/Yutta Roiza Bensinger - 2009 - 03.JPG: Copied (replaced existing)
2025/06/06 14:53:40 INFO  : Extended/Adler/Aryeh & Michal Bensinger/2009/Yutta Roiza Bensinger/Yutta Roiza Bensinger - 2009 - 13.JPG: Copied (replaced existing)
2025/06/06 14:53:40 INFO  : Extended/Adler/Aryeh & Michal Bensinger/2009/Yutta Roiza Bensinger/Yutta Roiza Bensinger - 2009 - 07.JPG: Copied (replaced existing)
2025/06/06 14:53:40 INFO  : Extended/Adler/Aryeh & Michal Bensinger/2009/Yutta Roiza Bensinger/Yutta Roiza Bensinger - 2009 - 11.JPG: Copied (replaced existing)
2025/06/06 14:53:40 INFO  : Extended/Adler/Aryeh & Michal Bensinger/2009/Yutta Roiza Bensinger/Yutta Roiza Bensinger - 2009 - 10.JPG: Copied (replaced existing)
2025/06/06 14:53:40 INFO  : Extended/Adler/Aryeh & Michal Bensinger/2009/Yutta Roiza Bensinger/Yutta Roiza Bensinger - 2009 - 19.JPG: Copied (replaced existing)
2025/06/06 14:53:40 INFO  : Extended/Adler/Aryeh & Michal Bensinger/2009/Yutta Roiza Bensinger/Yutta Roiza Bensinger - 2009 - 05.JPG: Copied (replaced existing)
2025/06/06 14:53:40 INFO  : Extended/Adler/Aryeh & Michal Bensinger/2009/Yutta Roiza Bensinger/Yutta Roiza Bensinger - 2009 - 09.JPG: Copied (replaced existing)
2025/06/06 14:53:40 INFO  : Extended/Adler/Aryeh & Michal Bensinger/2009/Yutta Roiza Bensinger/Yutta Roiza Bensinger - 2009 - 12.JPG: Copied (replaced existing)
2025/06/06 14:53:40 INFO  : Extended/Adler/Aryeh & Michal Bensinger/2009/Yutta Roiza Bensinger/Yutta Roiza Bensinger - 2009 - 15.JPG: Copied (replaced existing)
2025/06/06 14:53:40 INFO  : Extended/Adler/Aryeh & Michal Bensinger/2009/Yutta Roiza Bensinger/Yutta Roiza Bensinger - 2009 - 06.JPG: Copied (replaced existing)
2025/06/06 14:53:40 INFO  :
Transferred:        1.780 MiB / 1.780 MiB, 100%, 0 B/s, ETA -
Checks:             23400 / 23400, 100%
Transferred:           20 / 20, 100%
Elapsed time:         2.8s

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[avidocs]
type = crypt
remote = id:avidocs
password = XXX
password2 = XXX

[cgdocs]
type = crypt
remote = id:cgdocs
password = XXX
password2 = XXX

[famdocs]
type = compress
remote = famdocsenc:

[famdocsenc]
type = crypt
remote = id:famdocs
password = XXX
password2 = XXX

[fampics]
type = crypt
remote = id:fampics
password = XXX
password2 = XXX

[fonts]
type = crypt
remote = id:fonts
password = XXX
password2 = XXX

[id]
type = s3
provider = IDrive
access_key_id = XXX
secret_access_key = XXX
acl = private
endpoint = w1e0.va.idrivee2-52.com

[installers]
type = crypt
remote = id:installers
password = XXX
password2 = XXX

[music]
type = crypt
remote = id:music
password = XXX
password2 = XXX

A log from the command that you were trying to run with the -vv flag

The entire log is 23,457 lines long.
Rclone Sync Log

2

welcome to the forum,

  1. rclone check or rclone copy --dry-run on just one single file.
    use default values, no flags or absolute minimum flags to document the issue.
  2. post the full debug log.
3

Sure. Here is rclone sync on one file in the single problematic subdirectory. However, I can tell you now there won't be a problem as when I dropped use-server-modtime the files were not updated. Only on the full call above.

c:\rclone>rclone sync -vv "D:\Users\Parents\Pictures\Family Pictures\Extended\Adler\Aryeh & Michal Bensinger\2009\Yutta Roiza Bensinger\Yutta Roiza Bensinger - 2009 - 15.JPG" "fampics:Extended/Adler/Aryeh & Michal Bensinger/2009/Yutta Roiza Bensinger" --dry-run
2025/06/06 16:51:55 DEBUG : rclone: Version "v1.69.2" starting with parameters ["rclone" "sync" "-vv" "D:\\Users\\Parents\\Pictures\\Family Pictures\\Extended\\Adler\\Aryeh & Michal Bensinger\\2009\\Yutta Roiza Bensinger\\Yutta Roiza Bensinger - 2009 - 15.JPG" "fampics:Extended/Adler/Aryeh & Michal Bensinger/2009/Yutta Roiza Bensinger" "--dry-run"]
2025/06/06 16:51:55 DEBUG : Creating backend with remote "D:\\Users\\Parents\\Pictures\\Family Pictures\\Extended\\Adler\\Aryeh & Michal Bensinger\\2009\\Yutta Roiza Bensinger\\Yutta Roiza Bensinger - 2009 - 15.JPG"
2025/06/06 16:51:55 DEBUG : Using config file from "C:\\Users\\Avrah\\AppData\\Roaming\\rclone\\rclone.conf"
2025/06/06 16:51:55 DEBUG : fs cache: renaming child cache item "D:\\Users\\Parents\\Pictures\\Family Pictures\\Extended\\Adler\\Aryeh & Michal Bensinger\\2009\\Yutta Roiza Bensinger\\Yutta Roiza Bensinger - 2009 - 15.JPG" to be canonical for parent "//?/D:/Users/Parents/Pictures/Family Pictures/Extended/Adler/Aryeh & Michal Bensinger/2009/Yutta Roiza Bensinger"
2025/06/06 16:51:55 DEBUG : Creating backend with remote "fampics:Extended/Adler/Aryeh & Michal Bensinger/2009/Yutta Roiza Bensinger"
2025/06/06 16:51:55 DEBUG : Creating backend with remote "id:fampics/a9usrk179cg24iu0d5d8l2hqds/l8kfa461ckq2mr16d35c7l0t9g/5r3mnhda1cagatja9drut3estn7ia4519e0k9ob5ne5amopa672g/8g6c4tr5oshl1a00fle4ptoe6s/m8sbsft044okuti5nk39518gon84q5hcs1tfb2cbbkvvj1ofb0p0"
2025/06/06 16:51:55 DEBUG : Yutta Roiza Bensinger - 2009 - 15.JPG: Size and modification time the same (differ by 0s, within tolerance 100ns)
2025/06/06 16:51:55 DEBUG : Yutta Roiza Bensinger - 2009 - 15.JPG: Unchanged skipping
2025/06/06 16:51:55 NOTICE:
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         0.0s

2025/06/06 16:51:55 DEBUG : 4 go routines active

Here is the same call with all the options for comparison.

rclone sync -vvu "D:\Users\Parents\Pictures\Family Pictures\Extended\Adler\Aryeh & Michal Bensinger\2009\Yutta Roiza Bensinger\Yutta Roiza Bensinger - 2009 - 15.JPG" "fampics:Extended/Adler/Aryeh & Michal Bensinger/2009/Yutta Roiza Bensinger" --fast-list --use-server-modtime --fix-case --checkers 256 --transfers 32 --drive-chunk-size 256M --max-backlog=-1 --buffer-size 2G --dry-run

2025/06/06 16:52:59 DEBUG : rclone: Version "v1.69.2" starting with parameters ["rclone" "sync" "-vvu" "D:\\Users\\Parents\\Pictures\\Family Pictures\\Extended\\Adler\\Aryeh & Michal Bensinger\\2009\\Yutta Roiza Bensinger\\Yutta Roiza Bensinger - 2009 - 15.JPG" "fampics:Extended/Adler/Aryeh & Michal Bensinger/2009/Yutta Roiza Bensinger" "--fast-list" "--use-server-modtime" "--fix-case" "--checkers" "256" "--transfers" "32" "--drive-chunk-size" "256M" "--max-backlog=-1" "--buffer-size" "2G" "--dry-run"]
2025/06/06 16:52:59 DEBUG : Creating backend with remote "D:\\Users\\Parents\\Pictures\\Family Pictures\\Extended\\Adler\\Aryeh & Michal Bensinger\\2009\\Yutta Roiza Bensinger\\Yutta Roiza Bensinger - 2009 - 15.JPG"
2025/06/06 16:52:59 DEBUG : Using config file from "C:\\Users\\Avrah\\AppData\\Roaming\\rclone\\rclone.conf"
2025/06/06 16:52:59 DEBUG : fs cache: renaming child cache item "D:\\Users\\Parents\\Pictures\\Family Pictures\\Extended\\Adler\\Aryeh & Michal Bensinger\\2009\\Yutta Roiza Bensinger\\Yutta Roiza Bensinger - 2009 - 15.JPG" to be canonical for parent "//?/D:/Users/Parents/Pictures/Family Pictures/Extended/Adler/Aryeh & Michal Bensinger/2009/Yutta Roiza Bensinger"
2025/06/06 16:52:59 DEBUG : Creating backend with remote "fampics:Extended/Adler/Aryeh & Michal Bensinger/2009/Yutta Roiza Bensinger"
2025/06/06 16:52:59 DEBUG : Creating backend with remote "id:fampics/a9usrk179cg24iu0d5d8l2hqds/l8kfa461ckq2mr16d35c7l0t9g/5r3mnhda1cagatja9drut3estn7ia4519e0k9ob5ne5amopa672g/8g6c4tr5oshl1a00fle4ptoe6s/m8sbsft044okuti5nk39518gon84q5hcs1tfb2cbbkvvj1ofb0p0"
2025/06/06 16:52:59 NOTICE: Yutta Roiza Bensinger - 2009 - 15.JPG: Skipped copy as --dry-run is set (size 78.164Ki)
2025/06/06 16:52:59 NOTICE:
Transferred:       78.164 KiB / 78.164 KiB, 100%, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         0.0s

2025/06/06 16:52:59 DEBUG : 4 go routines active

Lastly, if it makes a difference, in the massive log file, only those 20 files had entries like:
2025/06/06 14:59:49 DEBUG : Extended/Adler/Aryeh & Michal Bensinger/2009/Yutta Roiza Bensinger/Yutta Roiza Bensinger - 2009 - 10.JPG: Computing md5 hash of encrypted source

No other of the 23,980 files had md5 hashes being computed. I though crypt folders didn't use hashes?

Thank you!

4

figure out which flag is causing the issue.

and post rclone lsl for the source file and dest file.

5

I’ll try to isolate the flag(s) over the weekend. In the meantime, would you have any thoughts about why those files are being hashed whilst none of the others are?