If you could minimize the number of flags you need to repro the problem that would be super useful.
I used a quite generic set of parameters, just --vfs-cache-mode writes so it might be something about your parameters.
Narrowing down the parameters would be most useful of these - I can try on my VM easily enough.
Ah, ha! Antiviruses do all sorts of strange things to windows file systems (to read and check the files before you execute them), and that would explain why it only happens on certain file extensions.
Can you try running without it?
Note that this issue has been reported before so it isn't just you!
Will do. I was basically just reusing a copy of one of my regular mounts. One of the things I have to simplify before I would ask someone external to reproduce it.
Will absolutely test.
It would fit the pattern of certain extensions being affected disturbingly well as AV probably filters on exactly this sort of thing as a security/performance compromise.
I swear - if it turns out to be Avira I am just about done with AV software in general....
@NCW Nick, you are an experienced developer - so let me ask you: Does software have a face? And if so, is there any method by which I would be able to psychically punch it? Because I have an urgent need to do this currently
So I checked out Avira and in it's real-time scanning section I found an "inclusion list" for filetypes....
"Oooooh Nooooooo..................."
Of course it's a perfect match to the failure-list... - and disabling RT scan immediately yielded correct results on all 17K files in my test-set once I reverted to rclone v1.49.4-003-gb5ea6af6-v1.49-fixes-beta, the recent go12 rebuild. (the latest workaround test-build still gave wonky results of only current dates, but that's likely redundant now).
I am doing one final full upload of the entire test-set to a real Gdrive remote for final confirmation (I did the initial tests to local remote in the interrest of time), but I would be extremely surprised if this final test wasn't successful. Real-time-scanning disabled obviously.
I profoundly apologize for making you spend so much effort on a rclone non-problem.
That said, if you said this wasn't the first time you have seen this issue reported then at least we learned something. Maybe this is worthy of a [PSA] or something along those lines, or in any case we now know the probable solution to other such reports in the future. So it's not a total loss I suppose - and I'm personally very pleased (if somewhat rage-filled still) to have this thorn in my side finally removed.
I will update this post only if final testing is not 100% correct, otherwise it can be assumed to be solved.
(I'll set it solved in 24hrs)
@NCW Amazingly I am still seeing modtime corruption here... using v1.49.4-003-gb5ea6af6-v1.49-fixes-beta and transferring the test-set to a real gdrive I still get a lot of [current date] on modtime, actually more than was corrupted earlier, presumably due to avira RTS.
Avira thoroughly uninstalled (because the damn thing auto-reenables RTS and will not allow permanent disable of the feature) and Windows Defender disabled in registry too for testing so it doesn't do something similar. The previous problem was very obviously Avira related, but there has to be more than one issue at play.
No obvious pattern on extensions now - not the same files being affected.
When quickly testing through VFS to local remote I don't get any errors though. Only on my "real use" test overnight.
I note a fairly clear pattern (although not 100% systematic) on alphabetic order (presumably the default order Windows iterates though when I copy a whole fodder of files). Starts with a clear pattern of corruption on every 16-19'th file. Through the night this gets gradually more and more frequent and ramps up to being above 50% towards the end of the transfer.
I will do some real thorough testing here on various factors - with and without the new workaround build and go to test on a clean VM if need be to track the error.
So you aren't getting the 1601 date but you are getting the current date? Is that correct?
That sounds like Avira was the cause of the 1601 dates (which makes sense) but there is something else going on! That something else is probably an rclone bug I would guess...
If you can post a log with -vv of it happening then it should be possible to figure out what is going on if it has a good one and a bad one in.
Correct. No 1601 or 8125 (or whatever that was) - only current dates when incorrect.
Note that this was with a regular beta build, not the workaround build. With the workaround build I was getting only current dates for some reason, so the first thing I did was try rolling it back to a pre-workaround version.
Which version do you prefer me to test (first at least) ?
The second time-debug-output one without the workaround fix in it?
I've seen quite a few bugs in rclone caused by AV stuff...