Best method for cloud-to-cloud server-side backup?

Help and Support
1

I have 2 gdrive accounts on different gsuite domains. I want to backup ~20TB from one gsuite account to the other. I tried using Google's built-in transfer function but it failed after a week and 4TB :rofl:

I want to avoid upload limits and just set it and forget it on my headless machine. This could take about a month. Here's what I plan:

  1. share folders from oldgdrive to newgdrive
  2. add server_side_across_configs = true and stop_on_upload_limit = true (just in case the upload hits a wall) to rclone.conf
  3. run rclone copy oldgdrive: newgdrive: -v -P --bwlimit 8M

As I continue to grow storage, I'll be able to run rclone sync oldgdrive: newgdrive: -v -P --bwlimit 8M which should hopefully make this easier in the future.

Is there anything I'm missing? Is there a faster/better way to do this? I tried using Shared Drives as suggested in the second link below but I couldn't copy the files, just transfer ownership.

References:

2

It's literally just rclone copy src: dest: --bwlimit=8M and let it go and come back in a month. Or you can get complicated and use service accounts and such.

3

Great, thanks. Any reason why the --bwlimit tag didn't work?

rclone copy oldgdrive: newgdrive: -v -P --bwlimit=8M

Transferred:      743.606G / 3.904 TBytes, 19%, 9.357 GBytes/s, ETA 5m47s
Errors:                 3 (fatal error encountered)
Transferred:          148 / 10156, 1%
Elapsed time:      1m19.4s

Failed to copy with 3 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
4

Ah server side copies. Since you're not any transferring data the bwlimit won't do it. You're not using any. I think you'd have to use stop_on_upload_limit as you mentioned and just rerun once a day.

5

Do you need help?
It can be done with service accounts easily
Will be over in an hour

6

Can you post a link describing what that is? I can probably figure it out.

7

Try any of these pls..
If not able to do..then ping me back
I can help

8

@Roshanconnor I'm using folderclone. In the multi-manager setup first step "Enable Drive API" I'm supposed to do that on the gsuite account that has gdrive files I want to transfer, right? The problem is my gsuite domain does not allow various IAM privileges, like resourcemanager.projects.get and serviceusage.services.get.

Am I SOL?

9

You dont have to use gsuite account
you can just use a normal gmail account
But remember, that gmail should have manager access to the TeamDrive

If you need more help..then pm me in Telegram