Is bisync ready for prime time?

(Note... This is repost from Bisync should be considered experimental so as to put my comments out to the broader community.)

I certainly agree with bisync being labeled as experimental.

Full disclosure - I wrote and maintained the rclonesync-V2 origin tool that bisync is a tight re-implementation of. rclonesync-V2 has been in wide use going on 5 years (since 2017), with 337 GitHub stars and certainly many more downloads.

My experience with rclonesync-V2, and more recently with bisync, says that the algorithm and safety features in bisync are pretty robust. If you are looking for a bidirectional sync utility, and understand the limitations, then I'd say give it a go. Start with an isolated directory tree and run with --verbose to view and understand all the transactions.

Some notable aspects of bisync that show a level of maturity of the tool

  • An extensive test suite covering lots of corner cases - allowing robust validation after code changes. This test suite was one of the main drivers of rclonesync-V2 being used as the reference implementation of bisync.
  • Safety feature in the bisync algorithm include:
    • A lock file prevents multiple simultaneous runs when taking a while.
    • Handles change conflicts non-destructively by creating ..path1 and ..path2 file versions
    • --check-access uses RCLONE_TEST files for checking for a linked/mounted portion of the filesystem being offline
    • --max-deletes check protects against one of the paths being offline being interpreted as all the files were deleted, or in case you've accidentally deleted a large directory on one side.
    • If something evil happens, rclonesync goes into a safe state to block damage by later runs, forcing user interaction and --resync to recover
    • --check-sync checks that both filesystems / paths actually match as a final step in a run. The default file comparison mode is by checksum. **
    • Detects changes to the user's --filters-file, and forces a --resync

bisync has several notable limitations that should be seriously considered before use. Also read the Troubleshooting section for particularly the --dry-run oddity and non-support for syncing Google Docs (& Photos).

I run several frequent syncs around my environment using these tools, and have for years. The problem I run into periodically is from trying to sync files that are open and being written to by some other application, which causes bisync to error out, and thus force a --resync. I have not had data loss issues since implementing the --check-access feature back in 2017.

** One personal usage note: I run bisync with --ignore-checksum to greatly speed things up. In this usage, file changes and sync checks are detected by timestamp only, which is fine for my use (and reliable). If you want, you can run a --check-sync only (without --ignore-checksum) periodically to force a checksum check.

So, in summary, I think this tool is pretty robust. It should work fine on most rclone supported backends (that support timestamps). bisync's safety features for avoiding data loss have several years of usage testing.

I certainly agree with bisync being labeled as experimental.

1 Like

I have added this note to the rclonesync README...

Public recognition for @ivandeex who spent more than a year porting bisync, and working through the rclone integration issues. Thank you very much @ivandeex !

I've been running a bisync beta every 5 minutes since November 2021, with no bugs. When the cloud server is offline, bisync gracefully fails and recovers without assistance with the server is back. I'm now porting my various local syncs over to bisync.

Hi, thank you for the amazing feature. I have been using it on a Mac without issues since its release.

One thing I want to report is that rclone bisync will not create ~/.cache/rclone/bisync automatically somehow. I have to create it manually and been working flawlessly afterwards. Rclone was installed from homebrew. The macOS was 12.3.

Again, thanks for the amazing feature.

Can you report a bug on GitHub about that please - that doesn't sound right

Sure, check out this link. Again, thanks for your amazing work.

1 Like

Whatever a CACHE_DIR path is on your OS, bisync will create CACHE_DIR/bisync automatically as confirmed by this code. The question is about default CACHE_DIR on MacOS, not about creating it.

The --resync option docs say this:
'This will effectively make both Path1 and Path2 filesystems contain a matching superset of all files. '
In other words no files would be deleted?

Mostly correct. However, during a --resync run:

  • Warning: Path1 files may overwrite Path2 versions.

In other words, if a different versions of a file exist on both paths, only the path1 version will survive.

Actually it did delete files. Did I do something wrong?

dt2:rclone bisync -iv --resync gdrive:Finance ~/HaackTrustGDrive/Finance --exclude '*~'
2022/04/13 14:38:03 NOTICE: bisync is EXPERIMENTAL. Don't use in production!
2022/04/13 14:38:03 INFO : Synching Path1 "gdrive:Finance/" with Path2 "/home/brad/HaackTrustGDrive/Finance/"
2022/04/13 14:38:03 INFO : Copying unique Path2 files to Path1
2022/04/13 14:38:13 INFO : Resynching Path1 to Path2
rclone: delete "RRH/2022-01-18 telecon notes.docx"?
y) Yes, this is OK (default)
n) No, skip this
s) Skip all delete operations with no more questions
!) Do all delete operations with no more questions
q) Exit rclone now.
y/n/s/!/q> 2022/04/13 14:38:59 INFO : RRH/2022-01-18 telecon notes Schwab.docx: Deleted

Hum... This I did not expect. I defer to @ivandeex.

Start with an isolated directory tree and run with --verbose to view and understand all the transactions.

Hopefully just the one file got clobbered.

Actually it did delete files

rclone bisync -iv --resync gdrive:Finance ~/HaackTrustGDrive/Finance --exclude '*~'
rclone: delete "RRH/2022-01-18 telecon notes.docx"

I defer to @ivandeex.

I tend to relate this quirk with docx files to

I think your best bet is to --exclude "*.docx" and see if it at least prevents data loss.

A full analysis will require full config and -vv dump from user. I am not requesting it right now because I will not have enough bandwidth to analyze/reproduce/fix in the short term. Re-deferring to yet unknown brave batmen from community.

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.