What does SetModTime Do?

Feature
1

just wondering what does feature, the documentation doesn't go very in-depth.

basically, i ran a sync to OneDrive, copied all the files on our server to a new server, and ran the sync again, and now rclone doesn't find any files that changed, but it suddenly wants to "set directory modification time" on all the cloud directories, even though their modification dates are all still the same in windows file explorer.

just wondering what that's about, because i haven't seen it before, and the SetModTime isn't a feature that was recently introduced.

jared

2

hi @jared, long time since we last communicated. hope all is well.

what version of rclone?

what version of rclone?

3

rclone v1.66.0

  • os/version: Microsoft Windows Server 2019 Datacenter 1809 (64 bit)
  • os/kernel: 10.0.17763.5830 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.22.1
  • go/linking: static
  • go/tags: cmount
4

so just to finish the story, we have an old server. we'll call it server A. i do a sync to a local backup, and then a sync to the cloud, onedrive in this case.

then after moving to the new server, server B, i do a sync and cryptcheck to the local backup, and all is well. no changes on the sync, no errors on the cryptcheck. however, if i do a sync to the cloud on server B, it wants to update thousands of directories with new modification times, and reupload thousands of files because the mod times vary by some milliseconds. but the most interesting part is, when i do a cryptcheck on server B and the cloud, it doens't find any differences

5

i don't expect full troubleshooting into this issue here. i'm just worried about what's going on, and don't like when i can't explain changes in thousands of files. you guys have all clearly dealt more with files, and understand more what's going on at the block level than i do. so i'm just wondering if this behavior makes sense to you. i'm happy to provide any logs you want though

6

copy just a single file using --dry-run and post the full log.

7

the files are all copied to the new server already, and it's in production. i'm happy to execute any command you like, but there's no files left to transfer

8

ok i should also mention, to transfer the files to the new server, i rclone synced to the backup server, then went to the new server and rclone synced from the backup server onto the new server

i just don't know enough about files to know why that would change the modified time a few milliseconds

9

fwiw, i would have just copied from old server to new server, without backup server in the middle.
i have never used rclone for local to local, so not sure exactly the unknown issues/quirks.
in addition, no mention of the operating system for all three servers.

OneDrive allows modification times to be set on objects accurate to 1 second and 10 ns on Windows

10

Your best bet.

Make a test case.

Create a file.
Do your process.
Share your commands a debug log.

Guessing what may or may not be the issue would be a futitle exercise as there are quite the permutations to play that game.

11

ok, here you go. i will redo a full run of the process. the reason i'm bouncing the files to the backup server enroute to the new server is because the production file server has many SMB shares with different permissions. but there's only one share that the backup account is dumping files into.

Old Server to Backup ->
here's the sync from the old server to the backup server, where the files obviously already exist because it's an active backup; just make sure i have a fresh sync (abridged because there were no changes):

2024/06/14 11:48:25 DEBUG : rclone: Version "v1.66.0" starting with parameters ["rclone" "sync" "--verbose" "--progress" "-vv" "E:\\Barbi" "Y:\\Server\\Barbi" "--log-file=C:\\Users\\Administrator\\Desktop\\Log.txt"]
2024/06/14 11:48:25 DEBUG : Creating backend with remote "E:\\Barbi"
2024/06/14 11:48:25 DEBUG : Using config file from "C:\\Users\\Administrator\\AppData\\Roaming\\rclone\\rclone.conf"
2024/06/14 11:48:25 DEBUG : fs cache: renaming cache item "E:\\Barbi" to be canonical "//?/E:/Barbi"
2024/06/14 11:48:25 DEBUG : Creating backend with remote "Y:\\Server\\Barbi"
2024/06/14 11:48:25 DEBUG : fs cache: renaming cache item "Y:\\Server\\Barbi" to be canonical "//?/Y:/Server/Barbi"
2024/06/14 11:48:25 DEBUG : Crocheted Rug.docx: Size and modification time the same (differ by 0s, within tolerance 100ns)
2024/06/14 11:48:25 DEBUG : Crocheted Rug.pdf: Size and modification time the same (differ by 0s, within tolerance 100ns)
2024/06/14 11:48:25 DEBUG : Crocheted Rug.docx: Unchanged skipping
2024/06/14 11:48:25 DEBUG : Crocheted Rug.pdf: Unchanged skipping
2024/06/14 11:48:25 INFO  : estelle frame: Set directory modification time (using SetModTime)
2024/06/14 11:48:25 DEBUG : Added delayed dir = "estelle frame", newDst=estelle frame
2024/06/14 11:48:25 INFO  : fonts: Set directory modification time (using SetModTime)
2024/06/14 11:48:25 DEBUG : Added delayed dir = "fonts", newDst=fonts
2024/06/14 11:48:25 INFO  : hoarfrost fr Kim: Set directory modification time (using SetModTime)
2024/06/14 11:48:25 DEBUG : Added delayed dir = "hoarfrost fr Kim", newDst=hoarfrost fr Kim
2024/06/14 11:48:25 INFO  : Jared & Barbi Final Photoshoot: Set directory modification time (using SetModTime)
2024/06/14 11:48:25 DEBUG : Added delayed dir = "Jared & Barbi Final Photoshoot", newDst=Jared & Barbi Final Photoshoot
2024/06/14 11:48:25 INFO  : Jared Barbi paperwork pics: Set directory modification time (using SetModTime)
2024/06/14 11:48:25 DEBUG : Added delayed dir = "Jared Barbi paperwork pics", newDst=Jared Barbi paperwork pics
2024/06/14 11:48:25 INFO  : JB + IJ wedding: Set directory modification time (using SetModTime)
2024/06/14 11:48:25 DEBUG : Added delayed dir = "JB + IJ wedding", newDst=JB + IJ wedding
2024/06/14 11:48:25 INFO  : JB wedding phone pics: Set directory modification time (using SetModTime)
2024/06/14 11:48:25 DEBUG : Added delayed dir = "JB wedding phone pics", newDst=JB wedding phone pics
2024/06/14 11:48:25 INFO  : passport photos: Set directory modification time (using SetModTime)
2024/06/14 11:48:25 DEBUG : Added delayed dir = "passport photos", newDst=passport photos
2024/06/14 11:48:25 INFO  : PRINT THESE BOYS: Set directory modification time (using SetModTime)
2024/06/14 11:48:25 DEBUG : Added delayed dir = "PRINT THESE BOYS", newDst=PRINT THESE BOYS
2024/06/14 11:48:25 INFO  : tauf cards 2024: Set directory modification time (using SetModTime)
2024/06/14 11:48:25 DEBUG : Added delayed dir = "tauf cards 2024", newDst=tauf cards 2024
2024/06/14 11:48:25 DEBUG : Thumbs.db: Size and modification time the same (differ by 0s, within tolerance 100ns)
2024/06/14 11:48:25 DEBUG : Thumbs.db: Unchanged skipping
2024/06/14 11:48:25 DEBUG : hoarfrost fr Kim/IMG-20240216-WA0002.jpg: Size and modification time the same (differ by 0s, within tolerance 100ns)
2024/06/14 11:48:25 DEBUG : hoarfrost fr Kim/IMG-20240216-WA0004.jpg: Size and modification time the same (differ by 0s, within tolerance 100ns)
2024/06/14 11:48:25 DEBUG : hoarfrost fr Kim/IMG-20240216-WA0006.jpg: Size and modification time the same (differ by 0s, within tolerance 100ns)
2024/06/14 11:48:25 DEBUG : hoarfrost fr Kim/IMG-20240216-WA0003.jpg: Size and modification time the same (differ by 0s, within tolerance 100ns)
2024/06/14 11:48:25 DEBUG : hoarfrost fr Kim/IMG-20240216-WA0003.jpg: Unchanged skipping
2024/06/14 11:48:25 DEBUG : hoarfrost fr Kim/IMG-20240216-WA0005.jpg: Size and modification time the same (differ by 0s, within tolerance 100ns)
2024/06/14 11:48:25 DEBUG : hoarfrost fr Kim/IMG-20240216-WA0008.jpg: Size and modification time the same (differ by 0s, within tolerance 100ns)
2024/06/14 11:48:25 DEBUG : hoarfrost fr Kim/IMG-20240216-WA0008.jpg: Unchanged skipping
2024/06/14 11:48:25 DEBUG : hoarfrost fr Kim/IMG-20240216-WA0015.jpg: Size and modification time the same (differ by 0s, within tolerance 100ns)
2024/06/14 11:48:25 DEBUG : hoarfrost fr Kim/IMG-20240216-WA0004.jpg: Unchanged skipping
2024/06/14 11:48:25 DEBUG : hoarfrost fr Kim/IMG-20240216-WA0012.jpg: Size and modification time the same (differ by 0s, within tolerance 100ns)
2024/06/14 11:48:25 DEBUG : hoarfrost fr Kim/IMG-20240216-WA0006.jpg: Unchanged skipping
2024/06/14 11:48:25 DEBUG : hoarfrost fr Kim/IMG-20240216-WA0002.jpg: Unchanged skipping
2024/06/14 11:48:25 DEBUG : hoarfrost fr Kim/IMG-20240216-WA0013.jpg: Size and modification time the same (differ by 0s, within tolerance 100ns)
2024/06/14 11:48:25 DEBUG : hoarfrost fr Kim/IMG-20240216-WA0009.jpg: Size and modification time the same (differ by 0s, within tolerance 100ns)
2024/06/14 11:48:25 DEBUG : hoarfrost fr Kim/IMG-20240216-WA0005.jpg: Unchanged skipping
2024/06/14 11:48:25 DEBUG : hoarfrost fr Kim/IMG-20240216-WA0015.jpg: Unchanged skipping
2024/06/14 11:48:25 DEBUG : hoarfrost fr Kim/IMG-20240216-WA0018.jpg: Size and modification time the same (differ by 0s, within tolerance 100ns)
2024/06/14 11:48:25 DEBUG : hoarfrost fr Kim/IMG-20240216-WA0012.jpg: Unchanged skipping
2024/06/14 11:48:25 DEBUG : hoarfrost fr Kim/IMG-20240216-WA0018.jpg: Unchanged skipping
2024/06/14 11:48:25 DEBUG : hoarfrost fr Kim/IMG-20240216-WA0013.jpg: Unchanged skipping
2024/06/14 11:48:25 DEBUG : hoarfrost fr Kim/IMG-20240216-WA0009.jpg: Unchanged skipping
2024/06/14 11:48:25 DEBUG : estelle frame/glaubensbekenntnis-Recovered.jpg: Size and modification time the same (differ by 0s, within tolerance 100ns)
2024/06/14 11:48:25 DEBUG : estelle frame/glaubensbekenntnis-Recovered.jpg: Unchanged skipping
2024/06/14 11:48:25 DEBUG : estelle frame/glaubensbekenntnis.psd: Size and modification time the same (differ by 0s, within tolerance 100ns)
2024/06/14 11:48:25 DEBUG : estelle frame/glaubensbekenntnis-Recovered.psd: Size and modification time the same (differ by 0s, within tolerance 100ns)
2024/06/14 11:48:25 DEBUG : estelle frame/glaubensbekenntnis-Recovered.psd: Unchanged skipping
2024/06/14 11:48:25 DEBUG : estelle frame/decaying-plaster-wall.jpg: Size and modification time the same (differ by 0s, within tolerance 100ns)
2024/06/14 11:48:25 DEBUG : estelle frame/decaying-plaster-wall.jpg: Unchanged skipping
2024/06/14 11:48:25 DEBUG : estelle frame/glaubensbekenntnis.jpg: Size and modification time the same (differ by 0s, within tolerance 100ns)
2024/06/14 11:48:25 DEBUG : estelle frame/glaubensbekenntnis.jpg: Unchanged skipping
2024/06/14 11:48:25 DEBUG : estelle frame/glaubensbekenntnis2222.jpg: Size and modification time the same (differ by 0s, within tolerance 100ns)
2024/06/14 11:48:25 DEBUG : estelle frame/glaubensbekenntnis2222.jpg: Unchanged skipping
2024/06/14 11:48:25 DEBUG : estelle frame/glaubensbekenntnis.psd: Unchanged skipping
2024/06/14 11:48:25 DEBUG : Jared Barbi paperwork pics/6J1A5010.CR3: Size and modification time the same (differ by 0s, within tolerance 100ns)
2024/06/14 11:48:25 DEBUG : Jared Barbi paperwork pics/6J1A5010.jpg: Size and modification time the same (differ by 0s, within tolerance 100ns)
2024/06/14 11:48:25 DEBUG : Jared Barbi paperwork pics/6J1A5011.jpg: Size and modification time the same (differ by 0s, within tolerance 100ns)
2024/06/14 11:48:25 DEBUG : Jared Barbi paperwork pics/6J1A5011.jpg: Unchanged skipping
2024/06/14 11:48:25 DEBUG : Jared Barbi paperwork pics/6J1A5010.jpg: Unchanged skipping
2024/06/14 11:48:25 DEBUG : Jared Barbi paperwork pics/6J1A5010.CR3: Unchanged skipping
2024/06/14 11:48:25 DEBUG : Jared Barbi paperwork pics/6J1A5011.CR3: Size and modification time the same (differ by 0s, within tolerance 100ns)
2024/06/14 11:48:25 DEBUG : Jared Barbi paperwork pics/Thumbs.db: Size and modification time the same (differ by 0s, within tolerance 100ns)
2024/06/14 11:48:25 DEBUG : Jared Barbi paperwork pics/6J1A5011.CR3: Unchanged skipping
2024/06/14 11:48:25 DEBUG : Jared Barbi paperwork pics/Thumbs.db: Unchanged skipping
2024/06/14 11:48:25 DEBUG : PRINT THESE BOYS/20230427_172628_0005-2.jpg: Size and modification time the same (differ by 0s, within tolerance 100ns)
2024/06/14 11:48:25 DEBUG : PRINT THESE BOYS/IMG_4757.jpg: Size and modification time the same (differ by 0s, within tolerance 100ns)
2024/06/14 11:48:25 DEBUG : PRINT THESE BOYS/IMG_4834.jpg: Size and modification time the same (differ by 0s, within tolerance 100ns)
2024/06/14 11:48:25 DEBUG : PRINT THESE BOYS/20230427_172940_0007-2.jpg: Size and modification time the same (differ by 0s, within tolerance 100ns)
2024/06/14 11:48:25 DEBUG : PRINT THESE BOYS/IMG_4834.jpg: Unchanged skipping
2024/06/14 11:48:25 DEBUG : PRINT THESE BOYS/Thumbs.db: Size and modification time the same (differ by 0s, within tolerance 100ns)
2024/06/14 11:48:25 DEBUG : PRINT THESE BOYS/20230427_172940_0007-2.jpg: Unchanged skipping
2024/06/14 11:48:25 DEBUG : PRINT THESE BOYS/IMG_4757.jpg: Unchanged skipping
2024/06/14 11:48:25 DEBUG : PRINT THESE BOYS/IMG_5566.jpg: Size and modification time the same (differ by 0s, within tolerance 100ns)
2024/06/14 11:48:25 DEBUG : PRINT THESE BOYS/20230427_172821_0006-2.jpg: Size and modification time the same (differ by 0s, within tolerance 100ns)
2024/06/14 11:48:25 DEBUG : PRINT THESE BOYS/IMG_5566.jpg: Unchanged skipping
2024/06/14 11:48:25 DEBUG : PRINT THESE BOYS/IMG_5388.jpg: Size and modification time the same (differ by 0s, within tolerance 100ns)
2024/06/14 11:48:25 DEBUG : PRINT THESE BOYS/IMG_4959.jpg: Size and modification time the same (differ by 0s, within tolerance 100ns)
2024/06/14 11:48:25 DEBUG : PRINT THESE BOYS/IMG_5388.jpg: Unchanged skipping
2024/06/14 11:48:25 DEBUG : PRINT THESE BOYS/Thumbs.db: Unchanged skipping
2024/06/14 11:48:25 DEBUG : PRINT THESE BOYS/20230427_172628_0005-2.jpg: Unchanged skipping
2024/06/14 11:48:25 DEBUG : PRINT THESE BOYS/20230427_172821_0006-2.jpg: Unchanged skipping

abridged

2024/06/14 11:48:25 DEBUG : PRINT THESE BOYS/IMG_4959.jpg: Unchanged skipping
2024/06/14 11:48:25 DEBUG : tauf cards 2024/followJesus1.jpg: Size and modification time the same (differ by 0s, within tolerance 100ns)
2024/06/14 11:48:26 DEBUG : Jared & Barbi Final Photoshoot/to print/wedding shoot/IMG_5528-2.jpg: Size and modification time the same (differ by 0s, within tolerance 100ns)
2024/06/14 11:48:26 DEBUG : Jared & Barbi Final Photoshoot/to print/wedding shoot/IMG_5532-2.jpg: Size and modification time the same (differ by 0s, within tolerance 100ns)
2024/06/14 11:48:26 DEBUG : Jared & Barbi Final Photoshoot/to print/wedding shoot/IMG_5528-2.jpg: Unchanged skipping
2024/06/14 11:48:26 DEBUG : Jared & Barbi Final Photoshoot/to print/wedding shoot/IMG_5532-2.jpg: Unchanged skipping
2024/06/14 11:48:26 DEBUG : Jared & Barbi Final Photoshoot/to print/wedding shoot/IMG_5458-2.jpg: Unchanged skipping
2024/06/14 11:48:26 DEBUG : Jared & Barbi Final Photoshoot/to print/wedding shoot/IMG_5549-2.jpg: Size and modification time the same (differ by 0s, within tolerance 100ns)
2024/06/14 11:48:26 DEBUG : Jared & Barbi Final Photoshoot/to print/wedding shoot/IMG_5550-2.jpg: Size and modification time the same (differ by 0s, within tolerance 100ns)
2024/06/14 11:48:26 DEBUG : Jared & Barbi Final Photoshoot/to print/wedding shoot/IMG_5549-2.jpg: Unchanged skipping
2024/06/14 11:48:26 DEBUG : Jared & Barbi Final Photoshoot/to print/wedding shoot/IMG_5464-2.jpg: Size and modification time the same (differ by 0s, within tolerance 100ns)
2024/06/14 11:48:26 DEBUG : Jared & Barbi Final Photoshoot/to print/wedding shoot/IMG_5566-2.jpg: Size and modification time the same (differ by 0s, within tolerance 100ns)
2024/06/14 11:48:26 DEBUG : Jared & Barbi Final Photoshoot/to print/wedding shoot/IMG_5561-2.jpg: Size and modification time the same (differ by 0s, within tolerance 100ns)
2024/06/14 11:48:26 DEBUG : Jared & Barbi Final Photoshoot/to print/wedding shoot/IMG_5566-2.jpg: Unchanged skipping
2024/06/14 11:48:26 DEBUG : Jared & Barbi Final Photoshoot/to print/wedding shoot/IMG_5561-2.jpg: Unchanged skipping
2024/06/14 11:48:26 DEBUG : Jared & Barbi Final Photoshoot/to print/wedding shoot/IMG_5574-2.jpg: Size and modification time the same (differ by 0s, within tolerance 100ns)
2024/06/14 11:48:26 DEBUG : Jared & Barbi Final Photoshoot/to print/wedding shoot/Thumbs.db: Size and modification time the same (differ by 0s, within tolerance 100ns)
2024/06/14 11:48:26 DEBUG : Jared & Barbi Final Photoshoot/to print/wedding shoot/IMG_5574-2.jpg: Unchanged skipping
2024/06/14 11:48:26 DEBUG : Jared & Barbi Final Photoshoot/to print/wedding shoot/IMG_5464-2.jpg: Unchanged skipping
2024/06/14 11:48:26 DEBUG : Jared & Barbi Final Photoshoot/to print/wedding shoot/IMG_5482-2.jpg: Unchanged skipping
2024/06/14 11:48:26 DEBUG : Jared & Barbi Final Photoshoot/to print/wedding shoot/IMG_5488-2.jpg: Size and modification time the same (differ by 0s, within tolerance 100ns)
2024/06/14 11:48:26 DEBUG : Jared & Barbi Final Photoshoot/to print/wedding shoot/IMG_5546.jpg: Size and modification time the same (differ by 0s, within tolerance 100ns)
2024/06/14 11:48:26 DEBUG : Jared & Barbi Final Photoshoot/to print/wedding shoot/IMG_5550-2.jpg: Unchanged skipping
2024/06/14 11:48:26 DEBUG : Jared & Barbi Final Photoshoot/to print/wedding shoot/IMG_5472-2.jpg: Size and modification time the same (differ by 0s, within tolerance 100ns)
2024/06/14 11:48:26 DEBUG : Jared & Barbi Final Photoshoot/to print/wedding shoot/Thumbs.db: Unchanged skipping
2024/06/14 11:48:26 DEBUG : Jared & Barbi Final Photoshoot/to print/wedding shoot/IMG_5488-2.jpg: Unchanged skipping
2024/06/14 11:48:26 DEBUG : Jared & Barbi Final Photoshoot/to print/wedding shoot/IMG_5546.jpg: Unchanged skipping
2024/06/14 11:48:26 DEBUG : Jared & Barbi Final Photoshoot/to print/wedding shoot/IMG_5472-2.jpg: Unchanged skipping
2024/06/14 11:48:26 DEBUG : Local file system at //?/Y:/Server/Barbi: Waiting for transfers to finish
2024/06/14 11:48:26 DEBUG : Waiting for deletions to finish
2024/06/14 11:48:26 INFO  : There was nothing to transfer
2024/06/14 11:48:26 INFO  : 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Checks:              1535 / 1535, 100%
Elapsed time:         0.9s

2024/06/14 11:48:26 DEBUG : 2 go routines active

transfer from backup server to new server:
Log2 Abridged.txt (88.2 KB)

i can see comments about delays. interesting...

12

that is what the v1.66.0 does, whereas v1.65.2 does not.

well, imho, windowz explorer is not a good choice, as it is limited, can display seconds, but not milliseconds/nanoseconds
whereas with rclone, default --modify-window on local windows file system can be as small as 100ns

13

I wonder why that is. cause I'm really paranoid when there's 2 million files at stake :no_mouth:

there's also the issue where the larger directories, now that they're on the new server, want to reupload thousands of files and hundreds of gigabytes because the mod time is a few hundred milliseconds different. and we've only done one of three servers.

I was wondering when that was gonna come up :face_with_hand_over_mouth:

but none of these transfers were done with windows explorer; everything was done by rclone. not sure if that makes a difference.

14

Try using --modify-window 2s - it sounds like the times got a little smooshed in the transfer process - that command will tell rclone you don't care if they are up to 2s out.

15

a little smooshed?! :scream::joy:
yes it makes almost all the difference. the most they're smooshed is around 900ms.

but how can a rclone transfer change that part of a file so randomly? it seems no file integrity has changed, cause I've done many hash checks and have found none so far.

I was blaming the new server, cause I was using what the industry professionals would probably call questionable practices: ESXi with PCIe passthrough 4 NVMe's to a Windows storage spaces array.

16

It is because you did the transfer A -> C -> B rather than A -> B and in one of those steps the modification time got truncated.

Not all backends have the same resolution of modification time.

It sounds like the timestamps got truncated to the nearest second somewhere in the chain.

The timestamp isn't part of the file - it is held separately by the OS and is normally known as metadata.

In summary - your data is fine. The timestamps are out by up to 1s. Either use --modify-window to ignore that or run rclone without it to correct the timestamps.

17

ok. thanks for clarifying. makes me feel better knowing i didn't potentially compromise a few million files :slight_smile:

18

hey @ncw, is this the reason why rclone 1.66 is paying more attention to mod times than previous versions of rclone?:

image
image1219×998 73.3 KB

did previous versions of rclone completely ignore mod times from directory metadata? merely curious :slight_smile:

jared

19

yeah. this must have been a bug that's now fixed. running the sync with --dry-run on 1.65.2 or 1.67 does not want to re-upload thousands of files/directories. switching back to 1.66 still does.

20
1 Like