Best method for cloud-to-cloud server-side backup?

I have 2 gdrive accounts on different gsuite domains. I want to backup ~20TB from one gsuite account to the other. I tried using Google's built-in transfer function but it failed after a week and 4TB :rofl:

I want to avoid upload limits and just set it and forget it on my headless machine. This could take about a month. Here's what I plan:

  1. share folders from oldgdrive to newgdrive
  2. add server_side_across_configs = true and stop_on_upload_limit = true (just in case the upload hits a wall) to rclone.conf
  3. run rclone copy oldgdrive: newgdrive: -v -P --bwlimit 8M

As I continue to grow storage, I'll be able to run rclone sync oldgdrive: newgdrive: -v -P --bwlimit 8M which should hopefully make this easier in the future.

Is there anything I'm missing? Is there a faster/better way to do this? I tried using Shared Drives as suggested in the second link below but I couldn't copy the files, just transfer ownership.

References:


It's literally just rclone copy src: dest: --bwlimit=8M and let it go and come back in a month. Or you can get complicated and use service accounts and such.

Great, thanks. Any reason why the --bwlimit tag didn't work?

rclone copy oldgdrive: newgdrive: -v -P --bwlimit=8M

Transferred:      743.606G / 3.904 TBytes, 19%, 9.357 GBytes/s, ETA 5m47s
Errors:                 3 (fatal error encountered)
Transferred:          148 / 10156, 1%
Elapsed time:      1m19.4s

Failed to copy with 3 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

Ah server side copies. Since you're not any transferring data the bwlimit won't do it. You're not using any. I think you'd have to use stop_on_upload_limit as you mentioned and just rerun once a day.

Do you need help?
It can be done with service accounts easily
Will be over in an hour

Can you post a link describing what that is? I can probably figure it out.


Try any of these pls..
If not able to do..then ping me back
I can help

@Roshanconnor I'm using folderclone. In the multi-manager setup first step "Enable Drive API" I'm supposed to do that on the gsuite account that has gdrive files I want to transfer, right? The problem is my gsuite domain does not allow various IAM privileges, like resourcemanager.projects.get and serviceusage.services.get.

Am I SOL?

You dont have to use gsuite account
you can just use a normal gmail account
But remember, that gmail should have manager access to the TeamDrive

If you need more help..then pm me in Telegram

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.