union remote keeps uploading ( Copied (Replaced Existing)) the same certain files over and over again. it must be failing to update certain aspects of the remote files, because it keeps thinking they are out of sync. this is not affecting all files, but the affecting ones remain an issue with each sync. on large syncs, this can get rather time consuming. also, i have noticed that if i run the sync over and over, the remote file and the local file have a modification time difference that's exactly the same each time. just as if rclone wasn't updating the remote file even though it gets uploaded and replaced each time.
Run the command 'rclone version' and share the full output of the command.
C:\rclone>rclone version
rclone v1.68.2
os/version: Microsoft Windows Server 2019 Datacenter 1809 (64 bit)
os/kernel: 10.0.17763.6054 (x86_64)
os/type: windows
os/arch: amd64
go/version: go1.23.3
go/linking: static
go/tags: cmount
Which cloud storage system are you using? (eg Google Drive)
Microsoft Onedrive Personal (combined with Union remote)
The command you were trying to run (eg rclone copy /tmp remote:tmp)
this has been a popular topic in the forum recently.
micro$oft onedrive personal has made some changes to modtime handling and so rclone has changed and is going to change again in the next version of rclone.
and also discussed at github.
check out my post
off-topic:
i have since upgraded to 2022 Datacenter, has been working great
but the change mentions the precision. why does the file still have the exact same modify difference each time there is a sync. it seems like it's not being updated on the remote side
interestingly, your post mentions that if the files are identical then rclone checks the hashes. is the reason i'm not seeing that on my particular sync that the files are not technically completely identical? (since their modify time differs a few nanoseconds)
gee. i hope you didn't use Windows Explorer in this upgrade process :o one of these months you should let me know how it's going otherwise. nice to see you're still monkeying around on the forum though
all of them are Onedrive Personal remotes. in the original log, you can see the list as rclone establishes a connection to each
yes i kind of abridged things by only posting one. i figured the interest would lie in the actual remotes rclone is connecting with using the Union remote. not so much the union remote itself.
FYI - there is potential source of confusion when dealing with OneDrive. There are two types of OneDrive Personal... like MS only upgraded some and not all. Or whatever reason they decided to use two different software versions.
Among multiple differences one is that they store mtime with different accuracy... some with 1s and some much more precise.
it would be less suspicous if each time there was a need for a sync, the modified difference would be a different number. but it's identical each time down to the 1/10,000th of a nanosecond. why doesn't it ever get changed?
yes. it updates the modification time, but when doing a subsequent sync, the modified time is still out by exactly what it was before the --refresh-times. would you like a log?
here is a log that shows rclone updating the mod time and saying "there was nothing to transfer". then, in the subsequent sync, it's back to the same game.
C:\rclone>rclone copy "E:\Bradley\Documents ATB" "monkeytest:Bradley/Documents ATB" -vv --refresh-times
2024/12/10 11:51:00 DEBUG : rclone: Version "v1.68.2" starting with parameters ["rclone" "copy" "E:\\Bradley\\Documents ATB" "monkeytest:Bradley/Documents ATB" "-vv" "--refresh-times"]
2024/12/10 11:51:00 DEBUG : Creating backend with remote "E:\\Bradley\\Documents ATB"
2024/12/10 11:51:00 DEBUG : Using config file from "C:\\Users\\Administrator\\AppData\\Roaming\\rclone\\rclone.conf"
2024/12/10 11:51:00 DEBUG : fs cache: renaming cache item "E:\\Bradley\\Documents ATB" to be canonical "//?/E:/Bradley/Documents ATB"
2024/12/10 11:51:00 DEBUG : Creating backend with remote "monkeytest:Bradley/Documents ATB"
2024/12/10 11:51:00 DEBUG : Creating backend with remote "MarianOnedrive:FS2/ęµēä¶éęé¤Ŗā¾ē½"
2024/12/10 11:51:02 DEBUG : Encrypted drive 'monkeytest:Bradley/Documents ATB': Waiting for checks to finish
2024/12/10 11:51:02 DEBUG : PO-3003.pdf: Modification times differ by -298.5971ms: 2024-08-15 09:25:48.2985971 -0500 CDT, 2024-08-15 14:25:48 +0000 UTC
2024/12/10 11:51:02 INFO : PO-3003.pdf: Updated modification time in destination
2024/12/10 11:51:02 DEBUG : PO-3003.pdf: Unchanged skipping
2024/12/10 11:51:02 DEBUG : Encrypted drive 'monkeytest:Bradley/Documents ATB': Waiting for transfers to finish
2024/12/10 11:51:02 INFO : There was nothing to transfer
2024/12/10 11:51:02 INFO :
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Checks: 1 / 1, 100%
Elapsed time: 2.0s
2024/12/10 11:51:02 DEBUG : 4 go routines active
C:\rclone>rclone copy "E:\Bradley\Documents ATB" "monkeytest:Bradley/Documents ATB" -vv
2024/12/10 11:51:34 DEBUG : rclone: Version "v1.68.2" starting with parameters ["rclone" "copy" "E:\\Bradley\\Documents ATB" "monkeytest:Bradley/Documents ATB" "-vv"]
2024/12/10 11:51:34 DEBUG : Creating backend with remote "E:\\Bradley\\Documents ATB"
2024/12/10 11:51:34 DEBUG : Using config file from "C:\\Users\\Administrator\\AppData\\Roaming\\rclone\\rclone.conf"
2024/12/10 11:51:34 DEBUG : fs cache: renaming cache item "E:\\Bradley\\Documents ATB" to be canonical "//?/E:/Bradley/Documents ATB"
2024/12/10 11:51:34 DEBUG : Creating backend with remote "monkeytest:Bradley/Documents ATB"
2024/12/10 11:51:34 DEBUG : Creating backend with remote "MarianOnedrive:FS2/ęµēä¶éęé¤Ŗā¾ē½"
2024/12/10 11:51:35 DEBUG : Encrypted drive 'monkeytest:Bradley/Documents ATB': Waiting for checks to finish
2024/12/10 11:51:35 DEBUG : PO-3003.pdf: Modification times differ by -298.5971ms: 2024-08-15 09:25:48.2985971 -0500 CDT, 2024-08-15 14:25:48 +0000 UTC
2024/12/10 11:51:35 DEBUG : Encrypted drive 'monkeytest:Bradley/Documents ATB': Waiting for transfers to finish
2024/12/10 11:51:35 DEBUG : ę»åøéēé¹ēé¹: Starting multipart upload
2024/12/10 11:51:35 DEBUG : ę»åøéēé¹ēé¹: Uploading segment 0/75234 size 75234
2024/12/10 11:51:36 DEBUG : PO-3003.pdf: quickxor = 840asdf089asdfpasdff46287 OK
2024/12/10 11:51:36 INFO : PO-3003.pdf: Copied (replaced existing)
2024/12/10 11:51:36 INFO :
Transferred: 73.471 KiB / 73.471 KiB, 100%, 73.464 KiB/s, ETA 0s
Checks: 1 / 1, 100%
Transferred: 1 / 1, 100%
Elapsed time: 2.9s
2024/12/10 11:51:36 DEBUG : 8 go routines active
based on your debug log. Modification times differ by -298.5971ms: 2024-08-15 09:25:48.2985971 -0500 CDT, 2024-08-15 14:25:48 +0000 UTC
the source file precision is smaller at 1ns, the dest file precsion is 1s.
so rclone cannot set the dest modtime to match the source modtime.
so you have three choices:
change the source file modtime to match the dest file modtime.
use --modify-window=1s or add that to the rclone config file.
wait for v1.69.0, which will default to --modify-window=1s
