Posting this as a followup request for the tail end of Getting a lot of "can't copy - source file is being updated" - #13 by ncw. I'm not certain if an issue was ever created on GitHub for it by DanG. I was not able to search and find it. Unless I'm looking in the wrong place, the source still only !oldtime.Equal(fi.ModTime())
I'm coming across ephemeral errors of modified source files during sync jobs:
Failed to copy: can't copy - source file is being updated (mod time changed from 2021-12-01 03:04:11.4318679 -0800 PST to 2021-12-01 03:04:11.4388747 -0800 PST)
The source filesystem is a Windows network share over a NTFS direct mounted drive. The discrepancy is 7ms which is surprising, but I'm far from an expert in how Windows is handing network shares.
I'd like to avoid using --local-no-check-updated as I think that could mask real issues. The sync is of snapshot data, so there shouldn't be any modification.
In the original thread, DanG's suggestion was to have a configurable level of tolerance in the modtimes. An alternative solution would be to provide a tolerance in the "freshness" of the file.
Original Suggestion:
oldtime = newtime +- tolerance
Alternative Suggestion:
newtime = now\checktime +- tolerance
Where newtime is fi.ModTime()
My rationale for the alternative suggestion is that I don't care if the file suddenly looks like it was modified way in the past. Only if the modtime is recent would there be a suggestion of write activity.
Correct me if I'm wrong Nick, but I thought that flag controlled comparison of modtimes between source and destination. What I'm encountering is a fluctuation of the modtime of the source.
Oh, I see what you mean! Sorry it was too late when I answered your question last night
These errors are fairly common on networked file systems and as you said --local-no-check-updated is the workaround for them.
You may need --modify-window as well depending on the precision of the destination.
I think you shouldn't be too worried about using --local-no-check-updated. It works very well for things like log files which are continually being appended to.
I guess what you are asking for is a --local-modify-window flag or something like that. This should probably adjust the precision of the local file system too so --modify-window isn't needed as well.
The default for this should be 0 as file systems should really always return the same modification time for files if they haven't been modified and the fact they aren't is a bug IMHO!
Do you want to have a go at adding this? It should be quite straight forward.
My concern with --local-no-check-updated is that it applies to the entirety of the job. While there may be some append only files for which it works, the flag would also clobber more complex file types. In my case, the files are a snapshot, so there should not be any modification of them whatsoever. Adding this flag removes this as a safety check.
I would absolutely be happy to take a stab at the code modification, but I'm not familiar with golang. Being able to read the logic of it based on familiarity with other languages is possible, but writing in an unfamiliar language is another story entirely. However, if I'm the only user who is looking for this functionality, I will happily put this on my list. This would come with the understanding that the timeline is undetermined though.
Could you elaborate on the use of "local" in the flag naming? I see how --modify-window without any scoping words should apply between source and destination. It seems to me --source-modify-window would be a more accurate description for this flag. I see the argument for keeping consistency with the --local-no-check-updated naming, but:
This flag functionally replaces the need for --local-no-check-updated. Setting --source-modify-window to a large value is effectively the same operation. However, some people may still prefer an explicit "no" flag.
I'd rename it to --source-no-check-updated instead
