I do not have a problem in itself. Just trying to get some guidance if possible. I am new to rclone, learned about its existence two weeks ago and I have been trying to set up the following configuration by reading the documentation and the forum. This is a diagram of what I would like to ultimately set up.
Before I start meddling with my wife's dropbox is the command and logic correct?
Since I only want 1 direction sync I am using sync but I do not truly follow the difference with copy, bisync, or mount. My understanding is that copy keeps the destination docs intact even if they are not in source, while sync removes them, bisync I understand is that it is literally what the name says sync but 2 directional. Mount I was not able to understand. Should I be using another one or sync is good?
There are many flags, am I using the right ones? I was not sure if I had to use --compare-dest
I reran sync multiple times deleting some documents in a test run just to understand a bit better. Is it correct to say that every time it runs it does not copy everything but only the files that have changed? I am asking because the "important document" took 25 minutes, the Media directory is larger, will it take so long every time it runs?
It is a lot of questions for one post:) But let's cover some.
Cloud storages were not created equal so there is no one fit all command/flags you can use. They have to be approached one by one - read docs carefully to understand requirements and limitation of each (as things like max path length or illegal characters are not the same).
Create specific appID registrations e.g. for gdrive. Similar step applies for dropbox and onedrive - all details are in docs. This is important to keep things work smoothly.
Both gdrive and dropbox are famous for very aggressive throttling policies you will need extra flags for these remotes e.g.:
gdrive: --fast-list
dropbox: --tpslimit 12
I would work here one by one and if you hit some problems create separate forum tickets.
If I was you I would use crypt remote everywhere otherwise your important_docs are all over the place in unencrypted format. It is your decision to have 100% trust in all these cloud companies but you also increase attack surface. What if one day data is stolen from one of them or your password compromised? etc. The best approach IMHO is TNO (trust no one). Encrypt/decrypt data locally and keep in the cloud only encrypted data. This is what crypt remote provides - please note that when configured it is transparent. You can forget it exist and all data is secure.
In terms of sync speed if all is configured correctly your initial sync will be limited by your Internet speed and later ones should be very fast - unless your data falls into some edge category like millions of small files.
Overall setup I am sure you thought a lot about what you need - there is no best solution here.
Me personally would use something different e.g.:
Synology probably already has Time Machine of your local computer so you have local backup
I would add TM style backup to the cloud (lets say to Onedrive) - I use program called arq for this but there are other options like restic/kopia/borg
to share data with wife I would use program called SyncThing - it quietly runs in the background and sync all changes immediately between your computers
I would use rclone (run from Synology) to replicate my onedrive cloud backup to another cloud provider or maybe also to Synology itself
I would use rclone to sync some important data to google drive (but with crypt)
Hey @kapitainsky, I appreciate you taking the time to answer everything and provide deeper context. Tonight I will look at everything in detail and try to make it work. A quick response to your suggestions:
Creating specific appID registrations. I have done it for gdrive and onedrive. I still have dropbox pending but these are actually very useful indeed.
Thanks for the --fast-list and --tpslimit 12 recommendations. I will read about it and include them
Good call using crypt. Better be careful than sorry.
Thanks for the recommendation on how you would set up everything (Time Machine, arq, SyncThing). I will look into those as well.
With regards to --compare-dest. The docs say "If a file identical to the source is found that file is NOT copied from source." I thought that this might help the sync speed and total time but was not sure.
Overall I have a lot of work to go over what you mentioned, I appreciate the time you took here.
This is absolutely my personal take:) But I see often people using rclone as a backup to the cloud. It is not. Think with your solution what happens when by mistake you delete some file on your computer. After some time it will be populated (deleted) from all your cloud storages. OK you use --backup-dir. But it is very primitive way preventing from only very obvious problems. If you lose/corrupt multiple files in multiple directories (maybe files with the same name) good luck in "getting back in time" which is what every good backup solution purpose is. Rclone is fantastic tool but not for everything.
It is weird option IMHO - there are many like this in rclone (it is nowadays big project with a lot of legacy things)- I think this one makes sense if you want to create some sort of crude backup. So for example I rclone sync source: dest:DAY0. Then next day I can create "delta" backup rclone sync source: dest:DAY1 --compare-dest DAY0. You think you want it use it:) But atm without providing DIR it does nothing. Even worse who knows what it does when DIR is nothing. It is quite common issue of software problems - users use options outside of the design scope. You would hope software check it and ignores... but it does not have to be the case. I always try to use only what is 100% required.
--compare-dest=DIR
When using sync, copy or move DIR is checked in addition to the destination for files. If a file identical to the source is found that file is NOT copied from source. This is useful to copy just files that have changed since the last backup.
You must use the same remote as the destination of the sync. The compare directory must not overlap the destination directory.
