Migrate from TeamDrive to Another TeamDrive

RobertusIT · September 21, 2019, 9:41am

What is the problem you are having with rclone?

I wish to copy, (not move), so get a backup of my Team Drive, to another Team Drive, server side copy.
Are in the same email, in shared drives.
In first the second Team Drive, was added in another email, so i added it to my first Team Drive, so i guess that isn't a problem.

I tried this command: rclone copy TEAMDRIVE1:Varie TEAMDRIVE2:Varie -v
But with this command, use my upload.

What is your rclone version (output from `rclone version`)

rclone v1.49.3

os/arch: windows/amd64
go version: go1.12.3

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Windows 10 64bit

Which cloud storage system are you using? (eg Google Drive)

Team Drive

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

I tried this command: rclone copy TEAMDRIVE1:Varie TEAMDRIVE2:Varie -v

A log from the command with the `-vv` flag (eg output from `rclone -vv copy /tmp remote:tmp`)

C:\Users\Robertus\Desktop\UP\RCLONE>rclone copy TEAMDRIVE1:Varie TEAMDRIVE2:Varie -v
2019/09/21 11:37:40 NOTICE: MYFILE.mp4: Duplicate object found in source - ignoring
2019/09/21 11:37:40 NOTICE: MYFILE.mp4: Duplicate object found in source - ignoring
2019/09/21 11:37:40 INFO : Google drive root 'Varie': Waiting for checks to finish
2019/09/21 11:37:40 INFO : Google drive root 'Varie': Waiting for transfers to finish
2019/09/21 11:38:39 INFO :
Transferred: 683.203M / 251.000 GBytes, 0%, 11.623 MBytes/s, ETA 6h7m35s
Errors: 0
Checks: 0 / 0, -
Transferred: 0 / 77, 0%
Elapsed time: 58.7s
Transferring:

MYFILE1.mp4: 9% /1.812G, 3.192M/s, 8m46s
MYFILE2.mp4: 8% /1.803G, 3.050M/s, 9m10s
MYFILE3.mp4: 2% /6.110G, 2.824M/s, 35m57s
MYFILE4.mp4: 1% /10.981G, 3.029M/s, 1h0m53s

Animosity022 · September 21, 2019, 9:50am

Server side across drives is off by default so you need to enable it via rclone config or add it into your rclone.conf manually:

https://rclone.org/drive/#drive-server-side-across-configs

RobertusIT · September 21, 2019, 7:23pm

Thanks, i tried without success this command

C:\Users\Robertus\Desktop\UP\RCLONE>rclone copy --drive-server-side-across-configs=true --stats 30s "TEAM_DRIVE1:Varie" "TEAMDRIVE2:Varie" -v
2019/09/21 21:21:05 NOTICE: MYFILE1.mp4: Duplicate object found in source - ignoring
2019/09/21 21:21:05 NOTICE: MYFILE2.mp4: Duplicate object found in source - ignoring
2019/09/21 21:21:05 INFO : Google drive root 'Varie': Waiting for checks to finish
2019/09/21 21:21:05 INFO : Google drive root 'Varie': Waiting for transfers to finish
2019/09/21 21:21:06 ERROR : MYFILE1.mp4: Failed to copy: googleapi: Error 404: File not found: xxxxxxxxxxxxxxxxxd1TOu., notFound
2019/09/21 21:21:06 ERROR : MYFILE2.mp4: Failed to copy: googleapi: Error 404: File not found: xxxxxxxxxxxxxxxxx1tyRh., notFound

Also without =true, so only --drive-server-side-across-configs, but same story.

Animosity022 · September 21, 2019, 7:27pm

Can you share your actual rclone.conf without passwords/keys?

RobertusIT · September 21, 2019, 7:32pm

[TEAM_DRIVE1]
type = drive
client_id = xxxxx.apps.googleusercontent.com
client_secret = xxxxxxxw6MPAm
scope = drive
token = {"access_token":"xxxxxxkxxxxxxxX2s__VxxxxxxxxxJg","token_type":"Bearer","refresh_token":"1/xxxa2Ixxxzxxxxxxxxxov_p","expiry":"2019-09-21T22:14:23.4246032+02:00"}
team_drive = xxxxxxk9PVA

[TEAMDRIVE2]
type = drive
client_id = xxxxxxxxxxxxx.apps.googleusercontent.com
client_secret = 1xxxxxxxx6gL
scope = drive
token = {"access_token":"xxxxxxxxxxxxxxxxxyVzPxxxxxxxxxxxxxxA","token_type":"Bearer","refresh_token":"1/xxxxxxxxxxxxxO0","expiry":"2019-09-21T22:17:48.3294604+02:00"}
team_drive = xxxxxxxxxxxxxxx

Animosity022 · September 21, 2019, 7:35pm

You may want to clean up the duplicates in your source and retry. Not 100% sure that's it or not though.

rclone dedupe the source.

RobertusIT · September 21, 2019, 7:39pm

2019/09/21 21:36:59 NOTICE: MYFILE1.mp4: Found 2 duplicates - deleting identical copies
2019/09/21 21:36:59 NOTICE: MYFILE1: Deleting 1/2 identical duplicates (MD5 "32cbcd894a6bf5fc510ae6b5eqc6fa5510")
2019/09/21 21:37:00 NOTICE: MYFILE1.mp4: All duplicates removed
2019/09/21 21:37:00 NOTICE: MYFILE2.mp4: Found 2 duplicates - deleting identical copies
2019/09/21 21:37:00 NOTICE: MYFILE2.mp4: Deleting 1/2 identical duplicates (MD5 "d6ee0cd225fbacwe517dd5662a8c0a166f")
2019/09/21 21:37:01 NOTICE: MYFILE2.mp4: All duplicates removed

done, i tried one more time to copy between drives, but same error

Animosity022 · September 21, 2019, 7:41pm

Just did a similar setup:

[felix@gemini rclone]$ rclone copy GD:hosts TEST: -vv
2019/09/21 15:39:33 DEBUG : rclone: Version "v1.49.3" starting with parameters ["rclone" "copy" "GD:hosts" "TEST:" "-vv"]
2019/09/21 15:39:33 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2019/09/21 15:39:34 DEBUG : hosts: Couldn't find file - need to transfer
2019/09/21 15:39:36 DEBUG : hosts: MD5 = 1450f7e2eb39701b01c3d307bf5c6a77 OK
2019/09/21 15:39:36 INFO  : hosts: Copied (server side copy)
2019/09/21 15:39:36 INFO  :
Transferred:   	       254 / 254 Bytes, 100%, 99 Bytes/s, ETA 0s
Errors:                 0
Checks:                 0 / 0, -
Transferred:            1 / 1, 100%
Elapsed time:        2.5s

2019/09/21 15:39:36 DEBUG : 7 go routines active
2019/09/21 15:39:36 DEBUG : rclone: Version "v1.49.3" finishing with parameters ["rclone" "copy" "GD:hosts" "TEST:" "-vv"]

How are you running the command? The conf you shared does not have server-side true in it.

RobertusIT · September 21, 2019, 7:45pm

I run ** rclone copy --drive-server-side-across-configs=true --stats 30s "TEAM_DRIVE1:Varie" "TEAMDRIVE2:Varie" -v**

from prompt in windows 10

thestigma · September 21, 2019, 7:49pm

--drive-server-side-across-configs=true
should work, but just to make sure that this is actually enabled - try setting it in your config instead. Use:

server_side_across_configs = true (note the different formating - copypaste it)
Put this in the bottom of each of your 2 remotes
save the config
remove " --drive-server-side-across-configs=true" from your command
then retry and see if the result is the same.

Are both of these drives accessibe under "shared drives" (previously called teamdrives) in the Google Drive website. In other words - is the user a member of both teamdrives?

I ask because it can be a bit confusing now to talk about shared files and shared drives which are fundamentally different things. Feel free to crop a small image from your Google Drive website that lists them just we can see this looks as expected.

Animosity022 · September 21, 2019, 7:51pm

The next thing to check would be permissions. The destination needs to have access to the source on the Share Drive to do the server side copy otherwise, it throws a 404 error as it cannot access the file.

Animosity022 · September 21, 2019, 7:52pm

It doesn't matter if you put it in the config or run the command line flag as they both do the same thing.

thestigma · September 21, 2019, 7:53pm

It shouldn't - no.
I'd probably still test it to be on the safe side since it fast and easy to do. Then at least we know it's not simply the function not being enabled properly.

EDIT: For example I'm not exactly sure if command-line parameters perhaps only apply to the source. I am also not sure if both drives need this feature enabled to work properly (I just always did it that way).
If the answer to both those questions are "yes" then it may not actually work (for both drives) when set via commandline.

RobertusIT · September 21, 2019, 7:58pm

I added server_side_across_configs = true on rclone.conf, remove --drive-server-side-across-configs=true from the command, run, but no way. Failed to copy: googleapi: Error 404: File not found: 1koRdiGTlRTspAhIo6TzXDeewJvU8K1tyRh., notFound

about permission, i add second team drive, like Responsable, the first option, i guess the higher option.

this guy on youtube, make it with rclone

thestigma · September 21, 2019, 8:06pm

In english:
For the source drive you need at least viewer (or any of the levels above).
For the destination drive you need at least "contributor" (or "content manager" or "manager", which are higher)

If you set both to "content manager" (full read/write/delete access) then you will definitely have no problems from these permissions.

You have to translate to your native languge as best you can from that
I expect the higher permissions are at the top of the list as they are in english.

I have no problems copying between my own teamdrives (on different domains) so it is definitely possible. We just have to identify the problem.

Animosity022 · September 21, 2019, 8:09pm

The 404 happens because of permissions. You'd need to turn them up so it works. You've already validated that information as you did a copy without the server side config and it worked.

The destination doesn't have enough permissions to the source.

RobertusIT · September 21, 2019, 8:12pm

WORKS!

my bad, because i tried to add a member in TeamDrive1, so i can see my shared drive, but needed also to add a member in TeamDrive2.

Now infact works like a charm!

Transferred: 251.000G / 251.000 GBytes, 100%, 6.713 GBytes/s, ETA 0s

Many thanks! I hope that account with this migration, stay on, and don't get a strike from google.

Thanks a lot for this amazing support.

unfortunately in other thread, my upload speed, is very strange, also via browser now, i'll change isp, when is possible ( if is the isp )

thestigma · September 21, 2019, 8:16pm

Nice! Yes, it's required that the users are all members of the drives they need to access between so that makes sense.

Google shouldn't care if you transfer server-side. That's perfectly allowed.
Do be aware that you still will need to obey 750GB/day limit.
After that rclone will start to get only 403 errors and you will need to wait until next day to continue.
Still ... it beats manual reuploading

RobertusIT · September 21, 2019, 8:22pm

When come up 750GB/limit, can i run the same command for the folder that i transfer now, and skip automatically all same files ?

Or i need a flag in command to avoid copy same files and waste limit day ?

thestigma · September 21, 2019, 8:27pm

rclone will automatically skip any files that are already on the destination (checks file + modtime). So no problem! You can stop and start as much as you want.

You could also use --checksum to compare hash value instead (but this is not possible if both drives are encrypted - with different crypt keys as hashes will not match). This is usually overkill.

You probably will want to add --fast-list to make the drive-listing faster - if you have many files. Google drive supports this. (don't worry if it takes about a minute before you see transfer starting this is normal with --fast-sync as it lists everything first).

I also recommend you add --progress (or -P) so you can see the progress of copying - how much is left ect.

--checksum (probably not needed)
--fast-list
--progress