Avoiding Bisync 'times are equal' conflicts with newer?

Love to hear that! :slight_smile:

The "real fix" was included in v1.69.1, so you should upgrade.

The bug was almost certainly a factor (especially as the 'fix' solved it for you). It remains generally true that --recover inherently carries a slightly higher risk of conflicts, but it may not have been responsible for the ones you experienced. It would be worth adding it back and seeing how it goes. (I have been using it myself, and have not seen problems so far.) My suggestion about removing it was mostly for debugging purposes, as it is much easier to debug if we can rule out extraneous factors and pare down to the simplest possible test case.

As far as order, I would probably start with --recover just because it is the most likely to cause conflicts (which is not to say that it necessarily will). If that goes well, you should be good to add the others back too.

I an not sure if this is related but I have been unable to successfully sync the last few days and I have not noticed any messages related cryptcheck before. About 30 minutes in it starts with this:

2025/03/04 13:43:36 DEBUG : code/newproject/services/webhook/tests/test.http: skipping equality check as size/hash definitely differ
2025/03/04 13:43:36 DEBUG : There are potential conflicts to check.
2025/03/04 13:43:36 INFO  : Encrypted drive 'idrive-sync-crypt:/': Crypt detected! Using cryptcheck instead of check. (Use --size-only or --ignore-checksum to disable)
2025/03/04 13:43:36 INFO  : Checking potential conflicts...
2025/03/04 13:43:36 DEBUG : Local file system at /home/salty/sync: Waiting for checks to finish

The next cryptcheck message is 15 minutes later, at which point it is repeated every second message and the sync slows down to a near halt of ~2 per second, the repeating cryptcheck message and another rechecking:

2025/03/04 16:13:39 INFO  : Encrypted drive 'idrive-sync-crypt:/': Crypt detected! Using cryptcheck instead of check. (Use --size-only or --ignore-checksum to disable)
2025/03/04 16:13:39 DEBUG : code/anothernewproject/blah/src/bytes.ts: rechecking

I have let it run for almost 10 hours and it has not complete.

I removed --recover (unsure if I have completed a sync since adding it a few days ago) and reverted back to the test rclone v1.70.0-beta test just in case but this made no difference.

Command:

rclone bisync "idrive-sync-crypt:/" "/home/salty/sync/" --check-access --fast-list --filter-from=/home/salty/.config/rclone/filters --links --error-on-no-transfer --conflict-resolve newer --resilient -vv

I could add in --size-only or --ignore-checksum as per the repeating message, or set the hasher back up without first testing for a time with just --recover added, but I wanted to check first as I never noticed this before, and the first messages suggest the encrypted drive is something the operation did not expect?

First things first, this message:

2025/03/04 13:43:36 DEBUG : There are potential conflicts to check.

means that the same file is new or changed on both sides, relative to the prior sync. Instead of automatically assuming this is a conflict, we attempt to compare the current file on both sides (using something similar to rclone check) to see if they are equal. This is because if they are equal (i.e. the same new version was independently copied to both sides), we can avoid creating a .conflict duplicate where it isn't actually necessary.

This works perfectly when both sides support hashes, but presents a bit of a problem for crypt, which does not support hashes. For example, if you run a normal rclone check on two remotes, where one of them is a crypt, you'll see you get a message like ERROR : No common hash found - not using a hash for checks. Since rclone check never considers modtime, it ends up essentially falling back to --size-only in this scenario. This is not ideal for our purposes, because it is possible for two files to have the same size but different content. (Technically this is also true for hashes like md5, but much less probable -- making hash a much better choice than size for equality checking.) Therefore, we don't want to assume that two files are equal just because they have the same size.

Rclone has cryptcheck for this purpose (robustly comparing crypt to non-crypt), and so bisync attempts to use it in this situation. This is what the following message means:

2025/03/04 13:43:36 INFO  : Encrypted drive 'idrive-sync-crypt:/': Crypt detected! Using cryptcheck instead of check. (Use --size-only or --ignore-checksum to disable)

However, an inherent side-effect of cryptcheck is that it has to basically read (i.e. download) the entire file on the non-crypt side and encrypt/hash it in memory on the fly. This can be quite slow, especially for large files. This is very likely why your sync slows to a crawl and doesn't complete for hours. This is also why you didn't experience this when using hasher -- because hasher can just use normal check with its stored hashes (which are very quick to retrieve).

Essentially, when bisyncing with crypt, users have to choose a tradeoff between:

  • Default (detecting false conflicts is highly accurate, but can be slow)
  • Use --size-only or --ignore-checksum (less accurate, but much faster)
  • Wrap your crypt in hasher (accurate and fast, but less convenient as it requires the hasher overlay)

Personally I go with the hasher option for my needs. But I can also imagine use cases where the default would make sense (ex. bisyncing a repo of text files where size is small but accuracy is paramount.)

1 Like

This has been working great, although I just experienced a random delete, of two files I created in the last few minutes, around the time I saw a rclone bisync run successfully complete.

I need to investigate this and provide more information when I can, but for now to just keep this thread from closing.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.