Hi all. I just discovered rclone and I am very excited to start using it. It has the potential to fix a data storage problem I thought was unfixable but I've encountered a few issues.
In short, I live in a rural area where the internet is very poor. If I'm lucky, I can get speeds of 25 Mbps download and 1.5 Mbps upload at best by combining two ISP's (and paying a fortune). This has made synchronizing my 2 TB of files a nightmare.
Before I moved here, I was on a cable internet connection and was able to upload all my files to Dropbox. Keeping up with the new files since then hasn't been a big deal since the bulk of it is already there. Fast forward a few years later and my Dropbox ran out of space so I decided to switch to OpenDrive which offers unlimited storage at a better price, which is great! But now my problem is getting those 2 TB worth of files copied from Dropbox to OpenDrive.
Ideally I would just use the clients and download the whole lot and re-upload it manually but that would take forever. Enter rclone which has has given me hope to transfer from one cloud (remote) to another and cut me out as the middle man. The problem here is that I can't seem to:
Tell rclone to use the servers bandwidth instead of my own (since it's so terrible and I need all I can get to do basic things)
Get it to transfer more than 100 kBytes/s on average. I can only assume this is because it's using my local bandwidth rather than Dropbox/OpenDrive's
I have tried using this command: rclone copy -vv --drive-server-side-across-configs=true "Dropbox:/example.mp4" OpenDrive:/ and a few variants but no luck. It transfers fine but the speeds are awful, again, likely because of the above I'm guessing, or there is a limitation somewhere on the cloud services.
Any suggestions/guidance here? What am I doing wrong? Is there a better way to do this? I feel like I'm overthinking it.
As far as I am aware at least, it's not possible to do a server-side transfer between completely different providers. They have to be on the same system - so for example one of Google's backends can server-side to other Google backends.
--drive-server-side-across-configs is a google-drive flag only (as the first part of the name suggests). It won't actually do anything here since neither remote is a google drive. But even one of them were in this case it would not help you. In that case it just defaults back to a standard "middle-man-transfer" and that is why your speeds are slow - you are piping it all through your local bandwidth.
Luckily, I can tell you a way to solve this anyway.
What you need for this job is a VPN, a virtual machine in the cloud. This can then act as your proxy. Then you simply use your config on that machine and do the transfer normally to leverage massive bandwidth.
In essence, any VPN can be used, such as one from Google Cloud - but the ideal scenario is to use a VPN in the same network as the destination. This is because ingesting data on a VPN is usually free or very cheap, but exporting costs more. I would check with Opendrive if they can provide a VPN. Even if they do not sell access to them (I don't know if they do but didn't see it on their frontpage) - they may be able to loan you one for the purpose of importing your data, as this is a very common way of moving large amount of data (and they are interested in your business presumably).
Even it turns out they can't - any VPN can be used. It will just cost slightly more than otherwise. 2TB isn't that costly though. Based on Google's pricing as a reference, it's typically 1-2 cents pr GB within the same continent (but it really depends on the providers pricing plan). The VM itself (a small one, which is all you need) typically runs for 1-2 cents an hour. The smallest ones can even be free at some providers.
The biggest challenge may be the technical side depending on your user-level. If you've never worked with a virtual machine before it may feel a little intimidating even if it's not really very difficult.
Thank you both for your quick and helpful replies. It makes a lot more sense to me now. I will look into VPN's/VM's and talk to OpenDrive as suggested. Thank you! Cheers
