Rclone Google Drive to Google Drive Migration for multiple users

Good day
I am new to Rclone and have spent hours reading through the forum for answers. I have two domains with G Suite installed on both - olddomainDOTcom and newdomainDOTcom.

I am using DOT to replace '.' as I am not allowed to post anything that looks like a link.

I have users in 'olddomainDOTcom' which I have created in 'newdomainDOTcom' and want to use Rclone to transfer their Google Drive data.

I have installed Rclone on my Mac and have created two remotes olddomain and newdomain.

I have configured server_side_across_configs on both remotes so I can impersonate users

For the olddomain, the command below shows me the Google Drive files and folders for the test1@olddomain.com user (source)
"rclone -v --drive-impersonate test1@olddomainDOTcom lsf olddomain:"

and the command
rclone -v --drive-impersonate test1@newdomainDOTcom lsf newdomain:" shows the Google Drive details for test1@newdomain.com (the destination)

So at this point, I believe my configuration is correct. I now need to copy the files from test1@olddomainDOTcom to test1@newdomainDOTcom

I have tried these two commands

rclone -P copy remote:test1@olddomainDOTcom olddomain: remote:test1@newdomainDOTcom newdomain:

rclone -vv -u -c remote:test1@olddomainDOTcom olddomain: remote:test1@newdomainDOTcom newdomain:

But none of them works. I will be grateful for any assistance.

Can you share your:

rclone version
Operating system

The command output with the -vv you have above.

Thank @Animosity022 I am using v1.54.0-beta.4766.75de30cfa
I was able to solve the copy issue.
In my configuration for the two remotes, I add options to impersonation. My remotes look like this.

type = drive
client_id = 10350942
scope = drive
service_account_file = /Users/albert/rclone2-289117953.json
server_side_across_configs = true
impersonate = test1@newdomain.com

type = drive
client_id = 102826185
scope = drive
service_account_file = /Users/albert/rclone-service-28ea65a3e.json
server_side_across_configs = true
impersonate = test1@olddomain.com

I was then able to use
'rclone copy olddomain: newdomain: --disable copy'
to copy files and folders from the Google Drive root for test1@olddomain.com to the root of test1@newdomain.com

Without adding the --disable copy flag, I experience an error for all the Google Docs, Google Sheets etc (basically anything created directly on Google Drive). This was the error message Failed to copy: failed to read description for Google Doc: googleapi: Error 404:

When I added the --disable copy flag, the Google created documents are copied not as Google documents but rather Microsoft Office files. So Google Sheets become Microsoft Excel.

Files such as Google Forms and Diagrams were not copied at all. I am guessing this is because --disable copy disables the server-side copying and the copying is done through my laptop which has no extension equivalent for Google Forms and Diagrams. I may be very wrong about this.

Now I am at a crossroad, I can copy from one Google Drive to another Google Drive but losing all Google documents. I have seen a lot of topics in the Rclone forum about this issue but have so far not seen any solution.

Rclone was trying to read the description for the file before copying it, but it got a 404 error when looking for the file.

404 error either means file not found, or peculiarly to google drive - you didn't have permissions to read the file.

Can you try this which prints an error but carries on if it couldn't read the description

v1.54.0-beta.4767.8ffbf2ef4.fix-drive-copy on branch fix-drive-copy (uploaded in 15-30 mins)

PS Are there any shortcuts involved with this?

With --disable copy rclone is exporting the docs and uploading them as docs. If you set --drive-import-formats then you can get rclone to re-import them as google docs. However a better solution would be to make it so you don't need to use --disable copy.

These don't have an export format so rclone can't export them and re-import them.

Because they don't have an export format, rclone ignores them normally so if you want to server side copy them (which should be perfectly possible) then that needs a fix.

Thanks, @ncw for the in-depth explanation, it is very eye-opening for a newbie like myself

Unfortunately, I don't know how to install the fix you have shared, the only way I know of installing Rclone is by running either
curl https://rclone.org/install.sh | sudo bash
curl https://rclone.org/install.sh | sudo bash -s beta
in terminal. I will check through the forum to see if I can find how to install a saved version on a mac.

I am going to try this.

If you download https://beta.rclone.org/branch/fix-drive-copy/v1.54.0-beta.4767.8ffbf2ef4.fix-drive-copy/rclone-v1.54.0-beta.4767.8ffbf2ef4.fix-drive-copy-osx-amd64.zip you'll have a .zip file.

Extract that zip file - it will have a directory in it called rclone-v1.54.0-beta.4767.8ffbf2ef4.fix-drive-copy

Open a terminal and cd into that directory. You can run that particular version by doing ./rclone when in that directory, or use the full path to that directory to run that particular version eg /home/User/ rclone-v1.54.0-beta.4767.8ffbf2ef4.fix-drive-copy/rclone

Hope that helps!

@ncw Thanks a lot for your assistance. I was able to download the file and ./rclone from the folder. But when I check the version with 'config version', I still get version v1.54.0-beta.4766.75de30cfa

Did you remember to do ./rclone version ? I checked the binary in the zip I linked and I think it is OK.

@ncw these are the outputs from using "./rclone version" and "rclone version"

Using ./rclone version
Alberts-MacBook-Pro:~ albert$ cd rclone-new
Alberts-MacBook-Pro:rclone-new albert$ ./rclone version
rclone v1.54.0-beta.4767.8ffbf2ef4.fix-drive-copy
os/arch: darwin/amd64
go version: go1.15.1

using rclone version
Alberts-MacBook-Pro:rclone-new albert$ rclone version
rclone v1.54.0-beta.4766.75de30cfa
os/arch: darwin/amd64
go version: go1.15.1
Alberts-MacBook-Pro:rclone-new albert$

So I support the new version is working.


Can you give it a test as above?

I have been able to test it. Here are the results:

(1) When user@olddomain.com had shared the Google Doc file with user@newdomain.com BEFORE the Rclone copy command, the Google Doc was copied. If server side is configured, the file was copied as a Google Doc format. if --disable copy is used, the file is copied as an MS Office format

(2) When user@olddomain.com had NOT shared the Google Doc file with user@newdomain.com, there was an error
"ERROR : Filename.docx: Failed to copy: failed to read description for Google Doc: googleapi: Error 404: File not found: 1al_2MmUu6kbH0g1oB3g0HnNgZU6LxkN39Hog., notFound
However the coping process continued, but the file was not copied at the destination.

So basically for Google Docs files, the receiving end should have sharing permissions for Rclone to copy it

Now the next thing I have to figure out is how to bulk change Google file permission for multiple users.

Thanks for testing. That is what I thought would happen but I wanted to double-check as the description reading coffee is new.

Was there no error from rclone if the copy failed? I suspect it would have been another 404 error.

Yes the receiving end needs to be able to read the files - if not you get the 404 error (rather than the 403 you might expect)

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.