Bisync question

Help and Support
12

Thank you so much for this additional info -- it was very helpful and I believe I have now identified the cause, and steps to reproduce it. It is indeed a unicode normalization issue, as suspected.

In short: the filename on your Path1 (macOS local) uses NFD, and the filename on Path2 (onedrive) uses NFC. (This is why your lsl on Path2 came up empty -- there's no file with that exact name, just one that looks the same to us humans.)

Path1 (macOS local - NFD):
300px-Schrödinger_cat.png
33303070782d536368726fcc8864696e6765725f6361742e706e67

Path2 (onedrive - NFC):
300px-Schrödinger_cat.png
33303070782d53636872c3b664696e6765725f6361742e706e67

This only makes a difference for the ö character. If we strip away all other characters, we have:

Path1 (macOS local - NFD):
ö
6fcc88

Path2 (onedrive - NFC):
ö
c3b6

(As background context: most of the world uses NFC, but macOS prefers NFD for some reason, and will sometimes (but not always) convert NFC filenames to NFD invisibly.)

Rclone's copy and sync commands (including the ones used by bisync under-the-hood) automatically normalize to NFC, unless --no-unicode-normalization is specified. This is why you can usually rclone sync between mac and non-mac without issue -- rclone will recognize the NFC and NFD versions as the same filename.

The trouble is that there are three places where bisync does NOT do this:

  1. When building the list of files that need to be transferred during --resync
  2. When building the list of deltas during a non-resync, but only for so-called "Unusual sync checks"
  3. When comparing Path1 to Path2 during --check-sync

#1 and #2 end up being harmless, because although it will try to copy the NFC and NFD versions as different files, the underlying copy function it calls is smart enough to realize they are the same file. It will therefore leave the files alone (if already equal) or update one of them (if not equal), but never create a duplicate.

#3 is the problem you discovered. Because --check-sync never normalizes, it thinks the NFC and NFD versions are different files. It therefore thinks something went wrong during the bisync run, because at the end of the run the Path1 file list does not exactly match Path2. This is why it throws the error.

Here is a bare-minimum case to reproduce:

% rclone touch NFD/ö 
% rclone touch NFC/ö
% rclone bisync NFD NFC --resync
2023/09/02 18:23:08 NOTICE: bisync is EXPERIMENTAL. Don't use in production!
% rclone bisync NFD NFC         
2023/09/02 18:23:17 NOTICE: bisync is EXPERIMENTAL. Don't use in production!
2023/09/02 18:23:17 ERROR : -          Path1 file not found in Path2       - ö
2023/09/02 18:23:17 ERROR : -          Path2 file not found in Path1       - ö
2023/09/02 18:23:17 ERROR : Bisync critical error: path1 and path2 are out of sync, run --resync to recover
2023/09/02 18:23:17 ERROR : Bisync aborted. Must run --resync to recover.
%

So, my opinion is that this is indeed a bisync bug (thank you for finding it!), and I will open an issue to track it. The long-term solution is probably that bisync should support normalization in the same manner as sync (i.e. always normalize when comparing between paths, unless --no-unicode-normalization has been specified.) In the mean time, as a short-term solution, I would recommend using --check-sync=false to bypass this issue.

Alternatively, you could convert all NFC filenames to NFD on both paths before using bisync for the first time. Assuming that Path1 is already NFD, and that Path1 and Path2 are already in sync, something like the following should do it (but I haven't actually tried it! Be sure to --dry-run first.)

rclone sync Path1 Path2 --no-unicode-normalization

A few other things worth mentioning in response to your comments:

Are you mounting the remote via rclone mount? If so, I'm not sure if I understand why you are also using bisync simultaneously. mount and bisync both do two-way sync, so it's typically only necessary to use one or the other, not both. And in fact, I'd recommend against using both, as it could potentially cause race conditions. You could find that they attempt to sync the same change at the same time, and whichever accomplishes it first confuses the other. This might well explain some of the other weird behavior you've been seeing.

To answer the question though: what I meant is "first time ever" (as between these exact paths), not "first time since mounting". (I must add: be sure to never bisync while unmounted -- it could cause serious data loss as bisync will think all the mount-side files have been deleted!)

Unless this is your "first bisync run" (as defined above), you should omit --resync here. The errors you cite are not caused by --resync, but by other things. The first error is caused by macOS automatically creating a .DS_Store resource fork file during your bisync run (this is the issue I mentioned in my first post). The second error is the NFC/NFD unicode normalization issue discussed above. Neither are directly caused by --resync (just masked by it.)

I see why that conclusion would seem logical, but it is correlation, not causation. It is not that the lack of --resync causes the problem, it's that once the problem exists, --resync corrects it (in the case of the .DS_Store error) or hides it (in the case of 300px-Schrödinger_cat.png.) It does so while also causing other problems (such as the potential for deleted files to get un-deleted), which is why it is not advisable to run --resync outside of the 3 specific scenarios I mentioned.

As above, I think this conclusion misidentifies the cause of the errors, and more importantly overlooks the dangers of using --resync on every run. To illustrate what I mean, try this simple experiment:

  1. Create a file named foo on one side.
  2. Run a --resync.
  3. Rename foo to bar on one side.
  4. Run a --resync.

RESULT: instead of both sides now having only bar, both sides now have BOTH foo AND bar. (Whereas, had you omitted --resync in Step 4, the rename would have worked as expected, resulting in only bar on both sides.)