What is your rclone version (output from
- os/arch: windows/amd64
- go version: go1.12.10
@NCW This is mostly intended for you, but I figure I might as well air the topic in public.
I would like to bring your attention to a fairly significant problem of modified dates being corrupted (effectively left blank and thus showing default values such as 1601 on Windows).
I touched on this subject last time you fixed the cache reference bug, but I haven't had time to delve into the detail until now.
I've needed some time to find what the triggers for the problem is, and I think I have found some good clues now that may give us something to go on in terms of debugging.
When uploaded via VFS write-cache, some files have their mod-time corrupted. the following steps appear to happen in those cases:
- File is copied to write-cache (with current date, which is wrong but this may be intentional?)
- The moment this finishes the file shows up on the mount as expected, but there the date is blank
- After the upload completes, some mechanism seems to trigger a re-download of the same file (some cache-refresh mechanism in the VFS?). This may be happening because it sees a mismatch between the dates I suspect.
- After file download the file in the cache now also has an empty moddate (it had current date before this)
2019/10/08 20:10:41 INFO : TESTING/test2/1GB.bin: Copied (new)
2019/10/08 20:11:57 INFO : TESTING/test2/1GB.bin: Copied (replaced existing)
Obviously this causes all sorts of havoc as modtimes are not only permanently lost (and relied upon by many rclone functions for proper operation), but also the cache is actually doing more harm to efficiency than not using it at all.
To contrast - On unaffected files this happens:
- file is copied to write-cache (with current date, which is wrong, but maybe intentional?)
- after landing in the cache it shows correct date on the mount (even though the date is wrong in the cache, so that's why I expect the above is not necessarily a bug)
- No apparent change when upload is complete
- File is not re-downloaded and everything seems to be ok.
There is also another layer to this as there seems to be a consistent pattern of certain file-extensions always being affected while others are never affected. This is based on limited testing and observations, but it is far too consistent to be random.
extensions that appear consistently affected in testing:
- .rar (but not other files in the same archive using 001, 002 extensions for example - indicating the extension is the trigger and not the file-format)
I suspect many more based on what patterns I see, but these are the ones I have tested properly.
A contrasting example of an extension I can not replicate the problem with (for replication of the issue) would be .bmp - this never fails under the exact same conditions.
- The issue does not seem to occur outside of the VFS layer - such as by an upload directly via console. Pretty sure it's a problem in the VFS, or happening in the VFS-to-Remote interactions.
- Could potentially be a Windows-specific issue just based on the lack of reports (can anyone confirm on Linux?)
- Pretty sure this is a fairly old problem dating back to at least 1.47-1.48 or more so I don't suspect recent changes - but I haven't done the detailed testing until now to be fair ...
If anyone has ideas on how to proceed or narrow down the problem even further - I am all ears
Full Debug log of issue as it is happening. 1 file "100MB.bin" was handled so you can search for that to make the job easier: