Get rid of rsync, replace it by rclone for remote backups

[silkyclouds]

Hi all,

So, I’m using rsync to backup my music to a remote gdrive folder, but to be honest, I do encounter a lot of issues.

1. For some reason, backups takes AGES. rsync barely reaches 2MB/s and transfer each file one after the other. if you count 4tb of “small” files, it literally takes 60 days to backup my full folder.
2. each time I relaunch rsync -a, it do re-scan my full folders, and files, which means it will start the transfer only after the scan is finished.

My goal is to do the following :

• I have two separate servers, which do contain a /music folder
• server one, is the “historical” one, which mean it currently do have 65k folders approx.
• server two, is a new server, and /music contains approx 2k folders
• remote gmail folder is also called /music

The content of server one is already synced using rsync to the gdrive.

Now, I have a server_one/music folder which is similar in content that gdrive:music

I’d like to run, lets say each night, a rclone sync task to :

copy ONLY the content which is NOT existing in gdrive:music, from server_one/music AND server_two/music

I’m quite afraid coz each time I read rclone sync it seems like running the job could delete files in my destination folder (gdrive).

So could someone get back to me with the right command which works well ?

Would make my day !

Thanks !!!

[calisro (Rob)]

You could just use copy all it doesn’t delete things. And if you want to ignore files that already exist (even if they are different) then there is a ignore-exisiting flag.

[silkyclouds]

@calisro, you are right. I’m now simply running a rclone copy --progress --size-only, and it analyses the destination folder, and only do copy files which doesn’t have the same size or doesn’t exist. works like a charm so far. and I’m reaching max speeds. Thanks for the advice !

[e12]

How do you " rsync to backup my music to a remote gdrive folder" - gdrive doesn’t support rsync, you are probably somehow mounting the gdrive and that’s slow. Also if the concern is with deleting the files there’s --backup-dir which saves the changed/removed files (and if you give it a different folder each time, like 2019-03-18-12-30-15 you have a very nice incremental backup with all versions saved).

[silkyclouds]

@e12 that’s right, my rclone remote is mounted using rclone mount.

but now I’m simply using rclone copy, and it simply works as rclone copy is by default not removing files from the dest folder I’m fine.