Sorry, I forgot about that... Thankfully I don't think you need it here.
If you really needed it I think it would be possible to work around though as I don't think it needs to be the same clouddrive - just the same logical remote. A lot of parts in more complex setups like this will be possible to greatly simplify once union gets it's overhaul, so I'd keep an eye on that. NCW indicated that with a little luck it might be a thing that can happen in one of the next 2-3 larger updates - and that's not very long into the future at the rate Nick runs this project 
Although the documentation says it must be the same remote, my experience is that it is not necessary. The above command works perfectly with three separate remotes/drives (local HDD, external USB and cloud remote:). Try it.
It also works if you specify a destination next to the source folder with both local and remote configs. Example:
rclone google1:Current google1:Diff --compare-dest=google2:Current
1 Like
Useful to know. Thanks!
That will probably come in handy at some point.
I use it to create aggregate and differential backups from Shared Drives to My Drive (useful, as my drive allows unlimited file counts).
You can of course sync directly from td: to mydrive: but it uses some of your 750GB quota.
Instead I do a differential sync using a service account with its own quota:
rclone td1:media td1:Diff --compare-dest=mydrive:media --drive-service-account-file=1.json --fast-list
Then a move:
rclone move td1:diff mydrive:media
The above goes server side and does not use any quota, as a move.
Then for content where I want to keep both a mirror and an archive I do a cleaner/separator that uploads no files but splits the mydrive content:
rclone sync td1:media mydrive:media --backup-dir=mydrive:backup/media --fast-list
In the end that keep mydrive:media identical to td1:media but stores all archived (unique name) files in backup/media.
NOTE: You could do all of that with a single command if you have small volumes. But the above allows doing backup chores without using inbound quota for mydrive.
SIDE NOTE: Diffmove does something very similar using rclone check. Created after a suggestion by ncw a long time ago (like his suggestion above with rclone lsf). Depending on your volumes/use-case diffmove can be faster or slower than the commands above. As union gets more powerful and the compare algorithms get more efficient I'm hoping diffmove won't be needed in future.
Nice script. That is functionality I've kind of wished was integrated into rclone. It's nice to have a bash script that does it, but would be even better if it could be done directly in rclone and thus be platform agnostic.
Thanks for sharing though, I already have some ideas on how I can use this to simplify some of my own scripted solutions. I'm less interested in the possible efficiency gains, and more interested in the flexibility it offers in automation and management.
1 Like
crontab, done.
sasync plus diffmove in crontab means I spend pretty much zero time with backups, mirrors and aggregation of content.
But as you noted, once union and compare-dest copy-dest get revved up a lot of things will be simplified. At the moment I find compare dest very handy for small-medium file-count libraries. But it seems to slow down massively with larger libraries.