I've been using Rclone for quite some time, but now I need to move some 700gb from Google Drive to another Gsuite account, I cannot use the copy feature it seems since the two accounts use two different domains
Which seems to indicate that the feature was implemented into Rclone, what exactly is the command? Also I intend on renting out a VPS to do this since my internet connection is not the best, would Rclone be downloading it? So I have to rent a VPS with 700GB in storage? Or does Rclone download, upload then delete the downloaded portion?
In this case this is no sweat and rclone can do this for you. For google to google you can make use of server-side copying and you won't even need to touch the data locally.
There is two ways to enable server-side copying. Either:
Set this in your config file under your drive remote: server_side_across_configs = true
or...
add this flag to whatever rclone command you run --drive-server-side-across-configs true
Either is fine - no need to do both.
Then you can simply use rclone copy (or rclone move if you prefer) and rclone will server-side the operation if possible. rclone copy OldGdrive: NewGdrive:\FromOldDrive -P
(-P here will enable some statistics to let you see the progress)
Server-side copies have nearly unlimited transfer speed, so they are very fast - but you will have to tell the server each file to copy, so the copy-time will mostly be determined by the sheer amount of files you have. The size of the files won't matter so much when you can copy them at gigabits pr second.
Using a VM in CGP like you mentioned is something that is often done, but usually only if you can't server-side it normally for whatever reason. For example if you need to import data between two completely different service-providers - or you not only need to transfer but also encrypt the data ect. In that case you don't need some huge storage in the VM no - a GCP microinstance will do, which would in practical terms be free for at least your first year (since you get free credits to use), but with careful use it would it would be free even past then - since there are certain things you can do within certain limits that remain free.
Let me know if you need me to elaborate on anything
The first year you get up to 300 virtual dollars to play around with in GCP, so that's there to experiment rather freely. Within that time you can certainly do a lot more than just a small microinstance - but that is typically all that is really needed for rclone-use.
(you have to actually activate the trial first of course )
As the error states - it is due to access permissions.
This usually means that you are trying to copy from Gdrive1 with user1 authentication to Gdrive2 with user2 authentication.
Usually the problem will go away if you also authenticate user2 to Gdrive1. Then then new drive is allowed to access these files.
If you didn't use server-side you would not see this problem because you would be the middleman, but with server-side there is no middle-man so there must be a shared authentication have permission to both read the old drive and write to the new one.
I hope that made sense? If not then I will try to clarify and step-by-step what you need to do.
I believe I understood so I what I did was move all the folders in my source drive to a single folder, then shared that single folder to the gsuite account I wish to send everything to, then I also shared the folder I wish to send it all to the source account.
Unfortunately still getting the error, what did I do wrong?
Sorry, I think I mixed up with some facts from another thread in the last answer and made some unfounded assumptions about your setup.
Let's try again - please start by confirming which version of rclone you are running. You can do this most easily by using the command: rclone version
The latest version is 1.49.5
If your version significantly older I highly recommend you upgrade https://rclone.org/downloads/
Because there have been multiple fixes not so long ago that correct exactly permission errors in server-side transfers. (I believe a few previously required restrictions were also removed).
Check that, report, and update if needed.
If that does not solve the problem can continue with something else.
What type of Gdrives are these?
Normal Gdrives, or "shared drives"? (previously called "teamdrives")
If you are unsure, they will be clearly labled on the google drive webpage.
Normal drives are just called "my drive" or something.
Shared drives are under a separate menu named as such.
One is a Google drive account which is my personal one, but it started getting too big so I went and got a Gsuite account which is the one I am trying to transfer the data to
Hmmm, I so rarely work regular drives so I'm trying to think how we would need to do it there. I think your shared folder tactic should generally work at least, so let's give that a try. (but I don't think it should be required when using the latest version).
When you did this - did you use the --drive-shared-with-me flag to get access to them? That may be needed. You probably need to copy them from user2's "shared with me" folder rather than using user1's files as the source in the copy operation.
(But I must admit I am a little confused as to why you managed to copy some files but not all).
Just so aren't making a simple mistake - tell me what method you used to activate server-side transfers. Did you use the flag method or the config method (and if so, on both remotes or just one)?
I used the config method, I did add it on both remotes since I was unsure
Apologies I am unsure on the rest, how I went about sharing these folders with each others account was going into the browser and right clicking them, then clicking on share and inputting the other account's email address
I think I just need to do a test real quick to see how this looks on a regular drive, because it's been so long since I tried (because I use primary teamdrives). Just want to make sure I'm not mis-teaching you here Be right back...
Ok, so here is how I just did a sucessful test between two of my regular free private Gdrvies:
On Drive1, using the googleUI, right-click on a folder (let's call this "testshare" for this example) and select share. Share it to the email address connected to your Drive2.
On Drive2, we now want to make a local copy of the stuff that got shared, and we can access this with by using --drive-shared-with-me. Let's use the following command:
rclone copy Drive2: --drive-shared-with-me Drive2:\localcopy -v -P
(I don't think you need to explicitly enable server-side copy in this sort of setup)
Give that a shot.
I am pretty sure that there is an easier way to achieve this than by using shared folders like this, but my unfamiliarity with the limitations of the regular drives has me a little uncertain. Anyway - this does work and is not hard, so it should suffice
Translated to plain english this would mean:
Take the files you find in the "shared with me" section of Drive2, and copy it to (the normal area) of Drive2.
The result of course is that you have a local copy of the data afterwards that is not shared, and is not dependent on the other account.
The key here is that the "shared with me" section is separate from the normal files area. You can see this visualized when you look at the google drive web interface. You see there is a special area called "shared with me". It is --drive-shared-with-me that lets us access this area that is otherwise normally hidden from rclone.
The reason for doing it this way is we ensure that Drive2 has explicit permission to access the files. Again, there should be easier ways of achieving this - but this does work (just tested it).
No problem!
When you have -v in the command it should say (server side copy) on the end of each file it copies. That way you know you are in fact moving them server-side. Also it should be quite obvious by the speed normally... at least fort larger files.
Hi Evernow and thestigma, Looks like you have it well sorted.
For future reference and possibly different use cases:
The OP said that he wants to move the content between two accounts on different domains. There are a couple of simple methods in addition to the solution you found above. Here is one:
From the Google Drive GUI drag folders into a master folder in your original account.
Share the master folder from the original account to the new account.
Go to Shared with me in the new account. Drag or right click the files/folders inside the master folder to a new location in the new account.
Unshare the master folder (if you wish). The files will remain in your new account, ready to use.
The principle advantage of copying the files with rclone is that ownership is set to the new account, whereas with the 'GUI drag' method the original account technically remains the owner. The downside is that you have to copy rather than move. For 700GB of files it's very quick, so the above method described by stigma works nicely. For 700TB this dragging method may be quicker.
For the typical user there is no problem having the old account continue to be the 'owner' of the files. However, if you want to transfer ownership there are a couple of simple things you can do:
Method1. Follow the GUI drag steps above, then
Use rclone to sync the files from the existing folder/files to a new folder (say, newfolder1). The newly copied files are owned by the new account.
Rename folder to oldfolder (or delete it) and rename newfolder to folder. This way you have full use of the files while the new copy is being made.
Method2. Follow the GUI drag steps above, then
If your new account is a personal/business gsuite then create a Shared Drive (formerly called Team Drives).
Drag your folders/files into the Shared Drive. Moving files into a Shared Drive automatically sets ownership to the account that owns the Shared Drive - your new account.
If you are happy to keep your files in the Shared Drive, you are done.
Otherwise...
Wait an hour (random period of time). Then drag the folders/files back to My Drive in your new account. The files will continue to be owned by your new account.
NOTE: Files can be dragged from My Drive to Shared Drives by any user. Folders can only be dragged by a Super Admin or a User to whom that right has been delegated. In most cases if you have your own GSuite then you are the de facto Super Admin. If not then you can ask your Super Admin to adjust the setting for you (or just use the rclone copy method). See https://support.google.com/a/answer/7374057
Method3. [[Only possible if your original account has Shared Drives and you have folder-drag permission.]]
Create a new Shared Drive (call it source1) in your old account and another Shared Drive (call it dest1) in your new account.
Share the Shared Drive source1 with your new account.
From the old account, drag the files/folders into the Shared Drive source1.
In the new account drag files/folders from source1 to dest1.
Done.
