Avoiding Bisync 'times are equal' conflicts with newer?

[saltysyncs]

Sorry it has taken me so long to come back to this. I am determined to resolve using rclone bisync for desktop sync and when I have I am very interested in using bisync: status GUI and notifications · Issue #7474 · rclone/rclone · GitHub instead of relying on my bash script and notify-send.

Recent errors (Bisync: Failed to read metadata: operation error S3: HeadObject & error reading hash from underlying) prompted me to purge the remote and for now I will only use the sync remote for my local code on my desktop, and for at least the next few weeks I have rclone disabled on my laptop. The smaller size and quicker runs should make it easier to find and address any issue. Ultimately even if I should migrate a lot of old personal documents I do not really need ready access to from bisync to a mount with rclone browser, I would like to sync more than code.

@nielash

As idrive uses s3, we also need to consider the s3-specific quirks related to modtime. (For example, rule out --use-server-modtime as a possible factor.) If possible, it would be really helpful to confirm whether we can reproduce this bug with a non-s3-based backend, to determine whether the problem is specific to s3 in some way. (For example, I bisync with drive and the only conflicts I ever see are the ones I expect to see.) Likewise, it would be helpful to remove hasher from the equation, if possible -- the goal is to narrow this down to the bare minimum repro case.

This sounds promising but as per the docs --use-server-modtime should probably be used with update and rclone tells me bisync does not support it. I tried without it in case it was not required with bisync for some reason but I cancelled it on the second run when as the docs warn it was copying and replacing all existing files on the remote.

Can you please make available a version that supports this, and if so can I just edit the command or is a purge and/or --resync required? I noticed when I tried just removing --use-server-modtime it started copying some files that should have already been uploaded to the remote so I purged and started over.

Also with hasher removed and --compare defaulting to size and modtime this is the command currently running in my script:

rclone bisync "idrive-sync-crypt:/" "/home/salty/sync/" --check-access --fast-list --filter-from=/home/salty/.config/rclone/filters --links --error-on-no-transfer --conflict-resolve newer --resilient -vv

When running with --update and --use-server-modtime I am unsure whether any --compare flags are required for local and server side checks, like to prevent bisync using normal modtime in addition to the server's, or so it will or will not not use size in combination with other checks?

This reads like checksum could this be useful for local side only comparisons but we did add the hasher so I guess that should be left out for now:

If --compare includes checksum and both remotes support checksums but have no hash types in common with each other, checksums will be considered only for comparisons within the same side (to determine what has changed since the prior sync), but not for comparisons against the opposite side. If one side supports checksums and the other does not, checksums will only be considered on the side that supports them.

@nielash

I think what would be really helpful here is to focus on the absolute simplest case -- the one where you are seeing unexpected conflicts even though:

• Only one machine is in use, and the other has been off for days
• --recover is not in use
• Graceful Shutdown is not in use
• We are certain the previous run finished successfully and the machine has not slept or otherwise interrupted a run since then.

I will confirm these before reporting any conflicts. I am curious though, are you personally using bisync only on a machine that is rarely off, or do you manually stop it or perhaps confirm a bisync successful before any shutdown?

I am trying to balance troubleshooting using the files in the remote normally, which seems to be the best way to trigger the issue! I should be able to keep doing this, only flagging conflicts that occur for the simplest case.

@nielash

Since idrive is cloud storage, not a machine, nothing should really be changing there unless you have told it to. So the next logical question is whether the file's content is really changing, or whether the change is just an illusion (i.e. the file on idrive is not truly changing, and bisync just thinks it is for some reason.) If the file is really changing, you need to figure out what is causing the change (like possibly some other app that is syncing changes to idrive, or edits made manually in the browser UI). If idrive gives you an event audit log for the file, that could be useful to determine this.

On the other hand, if the change is an illusion , then we need to determine why bisync thinks the file is changing when it is really not. It would help to know what specifically the change is (for example, is idrive reporting a modtime that is newer than the one we expect based on the last sync? If so, is idrive wrong, or is our expectation wrong?) The debug log can help you figure this out, as all changes should show the "old" and "new" values (with "old" being our expectation based on the last run.)

Until today I have exclusively used IDrive for rclone bisync though I just setup a new separate bucket and remote for a mount with rclone browser. I do not use their own desktop app or any other software to access IDrive, and I never make any changes using the Web UI other than during a purge, not least because everything is encrypted. It follows the likelihood is that the files are not really changing on IDrive - bisync just thinks they are.

I have versioning enabled in case it is something rclone can make use of, but I do not see any file history on the web app. I will see if there is some other way.

@nielash

The only valid values for --conflict-loser are num, pathname, and delete, so the above example would give you a syntax error. I think maybe you are thinking of --conflict-suffix, but this still only changes the leaf -- it does not move the file to a new directory. If I understand your goal correctly, you could accomplish something similar by using --backup-dir1 and --backup-dir2 in combination with --conflict-loser delete. But to be honest, I think you would be better off doing your testing in a new environment with dummy files, instead of in production with files you care about. I totally get the convenience factor since your existing setup appears to produce the bug easily -- but I think at some point the needs of your production environment will inevitably be at odds with the debugging goal of stripping this down to the fewest possible settings. And, ultimately we do need to find a way to reproduce this outside of your exact existing setup (because if we can't, then we haven't ruled out that something about your setup may be causing it.)

If this was something readily available it would make it easier to work and test for the issue at the same time but really the idea is a workaround for experiencing so many conflicts in the first place and resolving that will negate this.

@nielash

That's an interesting idea. To rephrase it, I think what you're suggesting (correct me if I'm wrong) is a new option that usually acts like --conflict-resolve newer when possible, but falls back to --conflict-resolve path2 as a tie-breaker when a winner cannot be determined by looking at modtime. I think this is worth considering -- feel free to submit a Feature Request for it on GitHub, if you would like.

Yes. As above this would be a way to manage the issue of conflicts occurring when I think they should not, as under normal circumstances it should be very rare the modtime is the exact same. I would still be uneasy of always deferring to a particular path especially as the current issue is that these conflicts are actually frequent!