Using Bisync with Dropbox while someone adds a file into a watched folder causes a critical error and a resync to be needed. Is this an expected behavior?

Help and Support
1

Background

We are a video production studio using dropbox to deliver final files to our client. We have been using a Synology NAS with Cloud Sync where every minute files are checked and synched between our watched Dropbox folder and our NAS. We have built a new system using TrueNAS Scale and are currently working on deploying it. For future reference, this machine is called "Webster", as is the new watched folder on Dropbox.

We also are aware of the problems and limitations with Dropbox, however even moving to another cloud service like Backblaze or AWS we would still want to have bisync work in the way we imagine. With an often-running service making sure both our local and cloud based storage are in sync.

Also yes, we are aware bisync is experimental. If pushed to it we could start to manually sync files, but a bidirectional sync is what we are used to.

What is the problem you are having with rclone?

When running a rclone bisync if someone adds a file into Dropbox while the command is running a critical "out of sync" error is thrown. If this is expected behavior, we are wondering how to make this work for our current workflow. More at the bottom.

Run the command 'rclone version' and share the full output of the command.

rclone v1.63.1

  • os/version: debian 11.5 (64 bit)
  • os/kernel: 5.15. 107+truenas (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: g01.20.6
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Dropbox

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone bisync dbox:/Webster /mnt/webster/core --exclude-from /home/admin/exclude-list.txt --transfers 10 --checkers 20 --fast-list --verbose --log-file="$LOG_PATH$LOG_FILE"

The rclone config contents with secrets removed.

[dbox]
type = dropbox
token = {our_token}

A log from the command with the -vv flag

admin@truenas[~]$ rclone bisync dbox:/Webster /mnt/webster/core --exclude-from /home/admin/exclude-list.txt --transfers 10 --checkers 20 --fast-list --verbose
2023/08/25 20:38:29 NOTICE: bisync is EXPERIMENTAL. Don't use in production!
2023/08/25 20:38:29 INFO  : Synching Path1 "dbox:Webster/" with Path2 "/mnt/webster/core/"
2023/08/25 20:38:29 INFO  : Path1 checking for diffs
2023/08/25 20:38:29 INFO  : Path2 checking for diffs
2023/08/25 20:39:29 INFO  : 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:       1m0.6s

2023/08/25 20:39:45 INFO  : - Path2    File is new                         - Miscellaneous/Komodo Test.mov
2023/08/25 20:39:45 INFO  : - Path2    File is new                         - Miscellaneous/WAN Show BTS Final Cut V4 Uncaptioned.mp4 (4K).mp4
2023/08/25 20:39:45 INFO  : Path2:    2 changes:    2 new,    0 newer,    0 older,    0 deleted
2023/08/25 20:39:45 INFO  : Applying changes
2023/08/25 20:39:45 INFO  : - Path2    Queue copy to Path1                 - dbox:Webster/Miscellaneous/Komodo Test.mov
2023/08/25 20:39:45 INFO  : - Path2    Queue copy to Path1                 - dbox:Webster/Miscellaneous/WAN Show BTS Final Cut V4 Uncaptioned.mp4 (4K).mp4
2023/08/25 20:39:45 INFO  : - Path2    Do queued copies to                 - Path1
2023/08/25 20:40:29 INFO  : 
Transferred:   	    1.750 GiB / 6.171 GiB, 28%, 39.810 MiB/s, ETA 1m53s
Transferred:            0 / 2, 0%
Elapsed time:       2m0.6s
Transferring:
 *                 Miscellaneous/Komodo Test.mov: 95% /941.330Mi, 19.736Mi/s, 2s
 * Miscellaneous/WAN Show…captioned.mp4 (4K).mp4: 16% /5.252Gi, 20.074Mi/s, 3m43s

2023/08/25 20:40:34 INFO  : Miscellaneous/Komodo Test.mov: Copied (new)
2023/08/25 20:41:29 INFO  : 
Transferred:   	    3.169 GiB / 6.171 GiB, 51%, 25.230 MiB/s, ETA 2m1s
Transferred:            1 / 2, 50%
Elapsed time:       3m0.6s
Transferring:
 * Miscellaneous/WAN Show…captioned.mp4 (4K).mp4: 42% /5.252Gi, 24.724Mi/s, 2m4s

During this file upload from Webster, we then copied a file from within Dropbox to the watched folder.

2023/08/25 20:42:29 INFO  : 
Transferred:   	    4.593 GiB / 6.171 GiB, 74%, 24.422 MiB/s, ETA 1m6s
Transferred:            1 / 2, 50%
Elapsed time:       4m0.6s
Transferring:
 * Miscellaneous/WAN Show…captioned.mp4 (4K).mp4: 69% /5.252Gi, 24.411Mi/s, 1m6s

2023/08/25 20:43:29 INFO  : 
Transferred:   	    5.947 GiB / 6.171 GiB, 96%, 21.432 MiB/s, ETA 10s
Transferred:            1 / 2, 50%
Elapsed time:       5m0.6s
Transferring:
 * Miscellaneous/WAN Show…captioned.mp4 (4K).mp4: 95% /5.252Gi, 21.432Mi/s, 10s

After this file finishes uploading you can see the actual error occur here:

2023/08/25 20:43:40 INFO  : Miscellaneous/WAN Show BTS Final Cut V4 Uncaptioned.mp4 (4K).mp4: Copied (new)
2023/08/25 20:43:40 INFO  : Updating listings
2023/08/25 20:43:41 INFO  : Validating listings for Path1 "dbox:Webster/" vs Path2 "/mnt/webster/core/"
2023/08/25 20:43:41 ERROR : -          Path1 file not found in Path2       - Miscellaneous/HFH Video-4K.mov
2023/08/25 20:43:41 ERROR : Bisync critical error: path1 and path2 are out of sync, run --resync to recover
2023/08/25 20:43:41 ERROR : Bisync aborted. Must run --resync to recover.
admin@truenas[~]$

We want to have the bisync command running on a regular basis so that if someone needs to pull a project from Dropbox onto Webster.Local then they can just move the folder and it will appear shortly after. Is there any way to force a resync run after an error like this is returned? It seems that running --resync for each of these commands is ill-adviside, but it is not clear to me why this would be the case. Can we run the command with --resync as a part of a recurring chron job? Or would there be a way to filewatch Dropbox and Webster and only sync when changes have occured on one or the other?

2

welcome to the forum,

i have never used rclone bisync, but perhaps this will spark an idea or solution.

rclone bisync --check-first

or

  1. rclone mount with --dir-cache-time set to a high value.
  2. pre-warm the vfs dir cache using rclone rc vfs/refresh recursive=true
    so that the mount will only see files currently in the vfs dir cache.
  3. use whatever bisync tool you want.

and fwiw, can learn more about the two vfs caches:

3

It is a known issue with the current version of bisync, and I'm working on a fix for it. Progress can be tracked in the following GitHub issues:

In the meantime, my recommendation is to use --check-sync=false and periodically do an independent integrity check as described here.

The reason this is ill-advised is because --resync will only copy (not sync) each side to the other. Therefore, if you included --resync for every bisync run, it would never really be possible to delete a file -- the deleted file would always keep reappearing at the end of every run (because it's being copied from the other side where it still exists).

While you could write a short script to do this, I wouldn't recommend it, as there are scenarios where a --resync would make things worse instead of better. The path1 and path2 are out of sync error can mean that files were added during the run (as you noted), but it can also mean that files were deleted (or renamed) during the run. In this case, running a --resync next would cause these intentionally deleted files to reappear. What you'd want to do instead (at least, until the underlying issue is more properly fixed in a future version) is manually delete these files from both sides, and then run the --resync.

In general, bisync requires a --resync as a safety switch when it thinks a manual intervention from the user could be required before the next run to avoid data loss. So, running a --resync automatically, without first checking things out, would essentially bypass this safety feature.

(In the current stable version, bisync is admittedly a bit overzealous about requiring --resync in some scenarios where it's not truly needed. This is addressed in the latest beta by the new --resilient feature.)

It's a good thought, but it will not help with this particular issue, because bisync has its own listing/delta engine that is not controlled by --check-first. The more relevant flag in this case is --check-sync.

1 Like
4

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.