I have a GSUITE account (DomainA.com) with multiple users that I'm moving to a new GSUITE account (DomainB.com).
I wanted to know if there's a way to feed rclone a CSV with multiple source email and their respective destination email and it would sync the file tree of GDRIVE?
I know i'll obviously need to authorize rclone and i have superadmin access to both GSUITE accounts.
It doesn't have to be done through CSV it's just an idea, if there are other ways through multiple commands or whatever, i'm open to all suggestions,
I don't mind running multiple instances of rclone (1 per user, etc..), I just can't find details on how to do it on behalf of other users than myself. (to send from userx on domainA and copy to userx on domainB)
I know how to sync rclone with 1 user as myself but not on how to do it for other users than myself.
Any hint in the right direction would be greatly appreciated.
Using the the --drive-impersonate flag with the correct service account credentials would work if you weren't using two drive remotes at once... The command line isn't clever enough to figure out which drive backend the flag applies to if you are using two, so it will apply it to both.
I think what you need to do is set up the two remotes with service account credentials - I'll call them domainA and domainB
You then need to tell rclone which users to impersonate - you can set up a remote for each user, or use environment variables, eg
With the way you're describing it, it seems that I would need to do one user at a time that way.
Would it be simpler/faster if for each user on the source DomainA, I setup a folder onto a single team share under DomainB? (I would create a folder per user and sync to their respective folders) and have them all dump automatically one after the other (this way i could run it overnight and all files and folders would be under the admin account team drive).
If I would create a remote per user (source), I could execute one command per user and they could all run in parallel correct?
I'm sorry if the following two questions are stupid:
What is the difference between the sync and copy commands?
When using sync/copy, does it download the files and then re-upload them or does it sync them between the accounts directly?
If I would create a remote per user (source), I could execute one command per user and they could all run in parallel correct?
Are you just asking if you can run multiple instances of rclone? If so, the answer is yes. They will not interfere with eachother. You can run rclone copy olduser1: newuser1: and then open a concurrent terminal session and run rclone copy olduser2: newuser2: and everything will be just fine.
Personally, I would run rclone copy olduser1: newuser1: -P -vv --log-file ~/user1copy.log so that you have a record of what was moved and where.
What is the difference between the sync and copy commands?
Where L is local and R is remote:
Copy copies files from L to R. That's it.
Sync makes R look like L. If something exists on L but not on R, that thing is copied to R. If something does not exist on L anymore, but exists on R, then that thing is removed from R. (Note changing things on R will not impact L)
When using sync/copy, does it download the files and then re-upload them or does it sync them between the accounts directly?
the --drive-server-side-across-configs should allow you to do this without downloading/re-uploading
I'm curious why you don't migrate everything using the gsuite tools, though. I don't know how large of a project this is, but anything more than a handful of users and I'm quite certain it would be simpler to migrate in gsuite, just based on setup time alone.
Because Google doesn't offer any tools to transfer Google Drive data without knowing the username and pwd for the user.
No, what I meant was, I'm trying to understand how Rclone works and what I need to do.
So I know i need to create two remotes with service account credentials.
What I'm having a hard time understanding is: as noted by ncw is I guess I can't use the "--drive-impersonate" flag because I have two remotes and i guess I can't define the impersonate flag to a specific remote through the CLI.
So I would need to do it through the config file hence why he provided this command.
I'm guessing you already have your second domain/gsuite account. And you likely are enjoying learning rclone. So ignore this if not helpful:
If your objective were to change domains but keep all of the same users and setup, gsuite has the capacity to add domains to an existing account. And to change the primary domain and/or delete the old primary domain.
Thank you for the info, for my needs these options don't work for me as I'm trying to merge domains onto a single new account. so I need to migrate the data from Drive one way or the other.
But it returns an error saying 401 not authorized.
when I created the remote config, I had added the client ID, left secret blank and provided a json path. After getting this error, I went ahead and deleted the ClientID line from the config, but still get the same error.
Not sure what's wrong. Any suggestions?
I'm gonna try and create a new client ID and re-insert the ClientID in the config, but any input in the meantime would be appreciated.
Thank you very much for all the help you've provided so far.
I'm trying to merge domains onto a single new account. so I need to migrate the data from Drive one way or the other.
That is exactly what the linked process will do.
I realize we are in the rclone forum, so this will be my last non-rclone related advice on the subject unless there is a specific question... but the above methodology really is the right way to do what you have described so far.
Nobody has discussed the transfer limits yet, so I should mention those as well.
yes, the third links does define how to transfer data and if you notice for Google Drive for data transfer as to be done through API or downloading and re-uploading (thing i want to avoid at all cost). Hence why i'm trying to use RClone to transfer the data from one domain to another.
so the "set" command for impersonate seems to have worked as I didn't get any errors when I executed it, simply went back to the command prompt.
When i execute "rclone lsf remote:" I keep getting error (pasted below), I tried editing the remote and putting the client id back in, same error.
C:\rclone>rclone lsf remote:
2020/01/24 14:11:57 Failed to create file system for "remote:": couldn't find root directory ID: Get https://www.googleapis.com/drive/v3/files/root?alt=json&fields=id&prettyPrint=false&supportsAllDrives=true: oauth2: cannot fetch token: 401 Unauthorized
Response: {
"error": "unauthorized_client",
"error_description": "Client is unauthorized to retrieve access tokens using this method, or client not authorized for any of the scopes requested."
}
yes, the third links does define how to transfer data and if you notice for Google Drive for data transfer as to be done through API or downloading and re-uploading (thing i want to avoid at all cost). Hence why i'm trying to use RClone to transfer the data from one domain to another.
Just to be clear -- I believe you are conflating accounts and domains. Moving from joe@old-domain.com to joe@new-domain.com is migrating a domain. Moving from joe@school.edu to joe@gmail.com after Joe graduates is moving an account. The first link explains how you can just turn joe@old-domain.com in to joe@new-domain.com without needing to copy anything whatsoever. You do not need to do the steps in the third link to make the data follow the user when migrating a domain.
Anyway--with that out of the way, it appears your service account isn't properly configured. Check the Admin Console and make sure it has access to the contents of the Drives. You will get unauthorized_client no matter what if your scope is wrong, even though that particular error sort of implies it's a token issue... so that's kind of counterintuitive.
Also make sure you aren't mixing regular user credentials and SA credentials in your rclone.conf.
Posting your rclone.conf would help (with redacted passwords/personal info). Based on the debug output you posted, this appears to be a gSuite configuration issue to me.
Replicated the error messages you were receiving just to be sure it was what I was thinking.
You haven't granted access to impersonate via the API to the service account you're using basically.
Impersoantion is a multi-step process, mainly covered in the guide, but might not be as concise....
Use current SA or make a new one Project owner is not needed DOMAIN-WIDE delegation IS needed
In order to add domain-wide delegation now you have to make the service account, and then you have to re-enter it for editing
Up top click edit
Middle of the screen between " Service account status" and "Keys" you'll see "Show domain-wide delegation"
Make sure that this is enabled! gives your SA a client-id {This is not the same as an OAUTH client-id}
This "Client-id" is what you will then enter Here
[navigation steps below]
Go to SecuritySettings in the admin panel. Third tab/card/whatever from the bottom should be "Advanced Settings" - when you click that you'll see a link to Manage API client access same as before, this is not the same as "APIs and Services"
In the first field enter that client-id you got once domain-wide-delegation was enabled.
Column label is " Authorized API clients"
Box label is "Client Name"
[disregard the "Example: www.example.com" part]
In the second field You have the enter the scopes you want to allow. IDK why I use the ones I list, but it works so eh...
Box label is "One or More API Scopes" PAY ATTENTION TO"(comma-delimited)"
I know what you're referring to and there's a reason why i can't change the domains, I really have no choice but to move them to a brand new account. I can't use any existing account (without going into the details), hence why the data has to be migrated. But that's outside the scope of this thread.
I'm not sure I understand what you mean by the service account having access to the contents of the drives.
I don't have an account for service account it's the super admin account i'm using.
As for the scope, there's only 1 address I added in there based on the config instructions:
2. Allowing API access to example.com Google Drive
In the next field, “One or More API Scopes”, enter https://www.googleapis.com/auth/drive to grant access to Google Drive specifically.
Also make sure you aren't mixing regular user credentials and SA credentials in your rclone.conf.
I don't have any credentials in the rclone.conf, I don't have access to it now to post it, I'll do it on Monday or Tuesday. I thought the whole point of putting a JSON is that you don't have to authenticate a user so credentials aren't needed.
What I do have in the conf under the [remote] is scope and type, JSON path, there maybe one or two lines missing (going off memory).
That was enabled, I do remember there's an option for apptype (might have the filed wrong here) and I left it blank based on the config instructions, and I remember reading somewhere that someone said to set it to "other" so not sure if this has an impact.
That I'll have to double check when i go back to work on Monday or Tuesday.
I only put 1 scope as per the instructions (pasted above)
I'll have to double check some of the info you guys have provided me,
Thank you for your input, i'll report back Monday or Tuesday after I've done my verification as I don't have access to those details outside of work.
