Continuous syncing

Help and Support
1

What is the problem you are having with rclone?

looking for a way to do continuous sync using rclone.

similar to the syncing clients that google, mega or yandex provide, but I find rclone more reliable - especially on linux. Additionally, rclone crypt is helpful.

I know rclone can only do 1 way sync, while they can do 2 way syncing - but will have to settle for that for now - if possible.

What is your rclone version (output from rclone version)

[adminuser@adminuser-pc ~]$ rclone version
rclone v1.49.5

  • os/arch: linux/amd64
  • go version: go1.13.1

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Linux Manjaro 64 bit

Which cloud storage system are you using? (eg Google Drive)

mega
yandex
google drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

looking for relevant command/script with guide. if it can be run from rclone browser or only command line.

2

Rclone isn't a sync tool, but offers a sync command.

You can use a cronjob and run it every 5 minutes or something like that and run a rclone sync to your remotes.

My upload script as an example does a basic check for if it's running and doesn't run again.

[felix@gemini scripts]$ cat upload_cloud
#!/bin/bash
# RClone Config file
RCLONE_CONFIG=/opt/rclone/rclone.conf
export RCLONE_CONFIG

#exit if running
if [[ "`pidof -x $(basename $0) -o %PPID`" ]]; then exit; fi

# Move older local files to the cloud
/usr/bin/rclone move /local/ gcrypt: --log-file /opt/rclone/logs/upload.log -v --exclude-from /opt/rclone/scripts/excludes --delete-empty-src-dirs --fast-list --max-transfer 700G

You can replace the move with sync and just run something like:

rclone sync /source remote:

Depending on the remote, there might be some performance tweaks but I'd start defaults and go from there.

3

Like Animosity says - all you really need to make a "continuous" sync is just to run the script from cron ever X minutes.

And while rclone does not have a prebuilt 2-way sync it is note hard to create one with and have it mimic the 2-way syncs that other software does.

A 2-way sync can be made in different ways though, and you have to decide how you'd prefer it to function.

rclone copy remote: /local
rclone copy /local remote:

That would make both places have all files if that's all you want.

It's wise to have something in the script that prevents multiple identical jobs from overlapping. Animosity shows and example in his script, and this example shows the same principle using a lockfile.

#!/bin/bash
#Lock the file so only one of these scripts can run at once.
lockfile -r 0 /tmp/uploadscript.lock || exit 1

#Upload command here

#Release lock
rm -f /tmp/uploadscript.lock

4

Pardon me, but Rclone is a synchronization tool. Including the slogan: "rsync for cloud storage"

As the most sophisticated uses are related to mount, etc., sometimes the sync function is relegated to the background, but Rclone is the only tool (AFAIK) that does it efficiently in a cloud environment.

1 Like
5

if you are going to experiement with rclone sync, it is easy to make mistakes and lose files.
make sure to check out the flag
--backup-dir

6

You parsed one bit which takes the sentence out of context. It isn't a sync tool but offers sync to cloud remotes.

A full sync tool is something like:

https://syncthing.net
or even actual rsync itself that offers full continually syncing capabilities.

rsync itself is very different from rclone.

7

i agree,
i never understood why rclone developers try to compare it to rsync.

sure, it is a nice slogan but not at all accurate.

8

thanks all.

I'll try as advised

9

And / or use --dry-run

Always be careful with experimenting with sync, because sync has the permission to potentially delete files. If you run it without being use what will happen, you could potentially lose data.

10

this wouldn't really work, as its not deleting older files for newer ones, nor resolving conflicts.

11

default behavior on copy is to update older files and not transfer over newer files - so that's covered. You would end up wiht only the most recently edited file in both places.
As for "resolving conflicts" you have to be a bit more spesific about what you mean.

This is just a really basic example. I'd have to know the exact behavior you wanted to see if you wanted me to give you something more specific. "2-way sync" is not strictly defined and you could interpret that in a variety of ways. It only tells me you want files to go both ways - not exactly what you want to have happen to them in when condition A, B or C applies.

The good thing about rclone is you have the flexibility to do basically any behaviour you want, as you have plenty of tools to work with. It would help to read up a bit about what the default behaviour of copy, move and sync is. Then we can always modify these with optional flags if needed.

12

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.