Is syncing files over multiple devices a supported use case by rclone?

Help and Support
1

What is the problem you are having with rclone?

I'm wondering if rclone can be setup so that files are synchronized across multiple devices, connected over ssh without worrying about data loss? By synchronizing I mean editing files on any one device including also deleting files and having this reflected reliably without data corruption on all devices. Can rclone mount accomplish that?

I've tried that with a linux desktop, wsl on windows and android over RSAF and had mixed results. There was some data corruption. Some files got completely wiped out (size: 0B) and some portions of files were temporarily or permanently lost . I do keep backups, so no worries. But I would like to know if what I'm doing is an "intended use case"™.

I've looked into rclone bisync, but wasn't sure if mounting a remote as a FUSE uses the same mechanisms?

If it's not supported I'm wondering if you've looked into unison or a similar solution focused solely on synchronizing folders? Would it makes sense in this scenario to couple it with rclone?

Run the command 'rclone version' and share the full output of the command.

rclone v1.66.0
- os/version: arch rolling (64 bit)
- os/kernel: 6.9.2-arch1-1 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.22.1
- go/linking: dynamic
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount remote: /path/to/mount \
--umask 077 \
--rc --rc-addr :<PORT> --rc-user <USER> --rc-pass 1234pass \
--cache-dir /path/to/cache/dir \
--vfs-cache-mode full \
--vfs-cache-poll-interval 5m \
--vfs-cache-max-age 8760h \
--vfs-cache-max-size 10G \
--attr-timeout 8700h \
--dir-cache-time 8760h \
--poll-interval 10s \
--multi-thread-streams 0

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[remote1]
type = crypt
remote = remote1-crypt:folder
password = XXX
password2 = XXX

[remote1-crypt]
type = drive
scope = drive
service_account_file = ~/.config/rclone/service-acccount-1.json
root_folder_id = XXX
shared_with_me = true

[remote2]
type = crypt
remote = remote2-crypt:folder
password = XXX
password2 = XXX

[remote2-crypt]
type = drive
scope = drive
service_account_file = ~/.config/rclone/service-acccount-2.json
root_folder_id = XXX
shared_with_me = true

[remote3]
type = drive
scope = drive
service_account_file = ~/.config/rclone/service-acccount-3.json
root_folder_id = XXX
shared_with_me = true

A log from the command that you were trying to run with the -vv flag

no log
2

No, they are totally different mechanisms (but should both be able to accomplish your goal.) Personally I prefer bisync (but I am biased.)

Bisync is essentially rclone's built-in equivalent of unison. There are some differences, but it should be the closest option to what you want.

If you are bisyncing or mounting more than two devices, be sure to set them up in a "star topology" where each spoke is synced with the same "hub" consistently (ex. multiple computers each synced to the same cloud drive, not to each other.)

1 Like
3

Thank you, that's very helpful. Do you know where I could find more information about differences between bisync and rclone mount? It would be good to know about any limitations or trade offs. There are many moving parts in my current setup and it's often hard to trace issues.

That's very interesting, I didn't think of using rclone in any other way. I've always assumed one central server that is publicly accessible (only google drive so far) any many "clients" behind NAT syncing back and forth.

What you are saying so far implies to me that it shouldn't be problem for such a star topology setup to work between many rclone instances (e.g. 4)? Even if one mix and matches bisync and rclone mount and uses encryption often on spotty connections?

If that was the case I'd assume that my previous issues were caused by other applications rather than the "rclone layer".

4

There is lots of info here:

In particular I would suggest reading the Limitations and Virtual File System sections of the mount docs.

There are many differences. In my opinion, the main two are:

  • Asynchronous (bisync) vs. more synchronous (mount)
  • Real files (bisync) vs. sparse / virtual files (mount)

In my opinion, that's asking for trouble... particularly with mount. Bisync has a robust system for handling sync conflicts, but mount doesn't really, so you'd run the risk of race conditions if you edit the same directory concurrently from multiple locations. The VFS adds a lot of complexity to mount...personally I've always found it a little too unstable (at least on macOS) to use for mission-critical syncing, but it can be a useful tool to browse a remote that is otherwise too large to download locally. For more critical and day to day tasks I prefer bisync and its built-in retries, conflict handling, and "real" files (especially as some of my apps don't play nice with sparse files). Just my opinion -- I'm sure there are others here who disagree.

5

Thanks again, that clears a few things up.

Sync conflicts were the main point of confusion for me, so it's great to see bisync introducing an explicit method to handle them. I assume that in rclone mount whichever clients syncs last is the one that simply overrides a conflicting file? I need to check how other more "popular" services adress it (like dropbox, google drive) although it's probably the same (whoever syncs last overrides a file saved by another user).

The main advantage of using rclone mount though is that syncing happens automatically as soon as files change on VFS. I assumed that bisync only syncs on a schedule? That seem like a trade off I'd be more willing to make at a cost of safety for personal use.

Not sure which solution would be more suited for a teamwork where you'd need both to ensure files don't get overwritten as well as close to instant synchronization. I've heard that people are able to use dropbox reliably although at work I only have experience with a samba/vpn setup when WFH or traveling.

6

In general that's right, but it's a little more complicated than that because of the variable --vfs-write-back and --vfs-cache-poll-interval duration:

Note that files are written back to the remote only when they are closed and if they haven't been accessed for --vfs-write-back seconds. If rclone is quit or dies with files that haven't been uploaded, these will be uploaded next time rclone is run with the same flags.

Also:

Using the --dir-cache-time flag, you can control how long a directory should be considered up to date and not refreshed from the backend. Changes made through the VFS will appear immediately or invalidate the cache.

--dir-cache-time duration   Time to cache directory entries for (default 5m0s)
--poll-interval duration    Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable (default 1m0s)

However, changes made directly on the cloud storage by the web interface or a different copy of rclone will only be picked up once the directory cache expires if the backend configured does not support polling for changes. If the backend supports polling, changes will be picked up within the polling interval.

So, there are a lot of factors that can determine who wins the "race".

Generally in the case of a true sync conflict, these services will keep both versions and rename them -- similar to what bisync (but not mount) does by default.

Sort of -- see note above about --vfs-write-back.

It is up to the user, but most commonly it is scheduled with cron. Personally I find the asynchronoous nature to be a feature, not a bug, because I don't necessarily want my file to start syncing while I'm in the middle of editing it -- doing it less frequently reduces a lot of risk of conflicts. I currently have my bisync jobs set to run every 10 minutes, which I find is about right for my workflow. And once in awhile when I need something synced sooner than that, it's easy to just run it manually. But there's no reason you couldn't schedule it more frequently than that if you wanted to (like every 1 minute).

Overall, I think mount wins on convenience, but bisync wins on safety. So it depends on what tradeoffs you want to make and also the nature of the files and the use case. For example, if you need several people to be able to edit different parts of the same document at the same time, probably neither tool is exactly right for that, and something like Google Docs might be your best bet.

1 Like
7

Thank you, that's really well put. It restores my confidence in using rclone and helps me appreciate its mechanisms much more (as opposed to just blindly copying rclone mount VFS file caching options). I'm still leaning more towards using rclone mount due to as you said convenience and being able to share links to files as soon as they are saved to disk. I will hopefully have a chance to use it in a teamwork setting soon. In a well-coordinated team I'm hoping this would be a more flexible solution than VPN+NFS.

I'm 99% as well that previous issues with seeming data loss were due to applications not dealing well with spotty connections (orgzly on android via rsaf). I was able to observe that these issues happened even when no other machine was using that remote meaning that it couldn't be some "race condition".

1 Like
8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.