I was wondering if there is anyone using rclone for backing up (as in copying) large Dropbox repositories (teamfolders). And if so, whether there are any specific gotchas or considerations that you can share that would be meaningful for me to consider before I start down this road.
I have successfully tested rclone out in a smaller scale, using it to create a copy of selected data from my dropbox into azure blob storage, and it ... well it just works as expected
So my question is not really if it works, but more if there are any experiences when it comes to doing this with really large amounts of data. And to quantify large, I am talking about maintaining a remote copy of 100's of TB of data, spread over 10's of million files and folders.
Do you think this is feasable?
And regarding the remote copy, any specific thoughts (pros) on using aws, azure, or something else?
I think the only gotcha might be if you hit dropbox rate limiting. Other users have suggested using --tpslimit 13 as the highest value which runs continuously but doesn't hit the rate limiter. That was for uploads, so not sure whether that will help for downloads.
Sure! Provided there aren't millions of files in a single directory which is rclone's weak point (unlikely with dropbox) then it should work fine.
Note there is a bug in the azureblob SDK at the moment which uses too much memory so you might run into that.
They are all good! If you want to save money then you can investigate B2 or Wasabi. Though I think if you use the Archive tier you can get down below the B2/Wasabi prices with AWS/Azure but watch out for the hidden costs!
hi,
there are so many options these days, no need to use just one service, just one rclone.
Wasabi - i have about 20TB in wasabi, a s3 rclone, known hot storage and fast api calls, no charges for ingress/egress - i use for veeam backups for disaster recovery.
then on a schedule, new backups goto wasabi and move some of that data to AWS deep glacier at $1.00/TB/Month.
Thanks for the quick feedback, that definitely tells me I should look deeper into this tool!
What about the conceptual setup, I have looked into the GUI (RCD) and using RC today as it would be valuable for me to be able to visualise the status / progress of the copy... but I get a feeling that this approach is a bit shaky sometimes, as I have experienced various inconsistent errors when submitting commands. (may be me trying things out though).
But thinking.... would the RC approach add any signigicant overhead, or would it be basically same as running the commands natively?
Ideally I would probably prefer just running the rclone commands in a few screen sessions (or similar), to keep things simple... but then I guess I would not be able to show progress in a webpage or similar, or is there another possible approach for this I can look into?
The RC is another way of accessing the same things as the command line so it won't be less efficient. It can be more efficient as it allows existing connections to be re-used.
You can show the progress via the --rc there is a nice javascript frontend (which I can't find right now) exactly for showing progress of transfers.
