Sync from one provider to another

I’m looking to sync my current encrypted google drive to another provider for backup purposes. But I’ve a couple of questions that I’m hoping someone can help clarify.

  1. If I run copy or sync from remoteA to remoteB will rclone download each file at a time to my server first, then upload it to the destination? (I’m assuming so but would be handy to clarify.)

  2. My google drive is encrypted. If i want the second remote to also be encrypted do I need to sync from encryptedA to encryptedB. Or can I sync remoteA to remoteB and use the same encryption credentials in encryptedA and encryptedB?

Yes it will.

I would sync the encrypted contents so remoteA to remoteB and use the same encryption credentials. That will mean rclone can use the hashes for data integrity and it will mean less CPU used for the transfer since it isn't encrypting and decrypting.

Or you can set up a cloud VM (GCP or other) and the transfer will be done extremely fast, and without donwload / upload.

Many thanks for the answers and the quick response!

Oh... now you have my attention! :smiley:

Unfortunately I'm not familiar with cloud VMs. (I'm assuming GCP is Google Cloud Provider?) If you have the time could you expand on this idea please @TowerBR?

EDIT: Is this reddit post pretty much what you're suggesting?

yes a Google compute. You can simply get a micro server which is free to use and sync both remotes if they are both Google...

If they are different you may need to worry about egress.

If they are both Google drive then you could also share the folder with the second remote and try server side transfers as well. ( --drive-server-side-across-configs with the newest beta.. yes you're still constrained to 750GB/day)

Thanks for taking the time. The other provider is Dropbox so I’m guessing that’ll cost me money.

My server on a gigabit pipe and currently transferring at 30Mbytes/s. (I think part of which is due to the flags I set in the sync command). I’ll see how it goes.

Yep! The free tier of Google computer is enough.

If I understood correctly you want to transfer from Google Drive to Dropbox. You will not have egress charges, only have the limit of 750 Gb per day, as @calisro said).

How will he not have egress charges? He'll be pushing data to Dropbox.

only have the limit of 750 Gb per day

Isn't this limit related to uploads to google drive or is that how much bandwidth the free tier of google compute offers?

Related to uploads to GD.

Egress charges from Google Drive. Other services like Backblaze B2 have egress charges. Am I missing something?

And I think since he's moving to Dropbox, he already has a service signature ... (unlikely that he will use less than 2 GB, the free level)

Yes he's moving to dropbox. Which means he'll be making connections from gc to Dropbox. Those connections will incur significant egress charges.

Ah, okay, really.

But I think the $ 300 USD trial credit is enough for a good amount of data transfer. It all depends on how much data he wants to transfer.

Right. $300 may cover up to ~2TB (.11 per GB EGRESS). BUT I'd be really careful doing this. Charges can add up quick. I was mentioning that even without that $300 credit a simple free tier on micro can transfer from Google to Google without any charges since it's all on the Google side.

As soon as you attach an external IP address to a VM in GCS that's when you have to be really careful about egress as even "internal" Google services can spill to egress because your default route becomes external.

Back on topic though. I'd stick with a regular VM to transfer to Dropbox...

1 Like

Maybe I’m ignorant here but my dedi server has a gigabit connection. Is cloud compute gonna be that much faster? I guess it removes the need to download?

To answer the question of how much data - I’ve around 10TB in drive.

Also thanks both for getting involved and offering your advice, really appreciate it. :+1:

Nope. Just use your server. You're limited to 750gb a day anyway.

You can use Feralhosting.com's cheapest tier (10 pounds per month) - they do not rate limit, no data caps, and I used it to sync from Dropbox to Drive. Much cheaper than anything you'll find with AWS or GCloud.

Feral servers are much faster than 1 gigabit, they have multi-20GBit peered connections. If you're going from Drive to Dropbox, it'll take no time at all to sync your files.