Transfer 2TB of data from a Google Drive Edu account into Google One

asdffdsa · November 10, 2022, 9:40pm

well, if that is all there is, then you are correct.

tho, the command should fail with 404.
i tried the command and it did fail with 404.
need to set the correct permissions and that is the OP issue here, am i correct?

Animosity022 · November 10, 2022, 9:48pm

Unknown as I'm not sure what's been done or not done.

Best to just try it out with a small folder and go from there with some output to look at.

asdffdsa · November 10, 2022, 10:25pm

not an expert with gdrive like you, but i did try, does not work, get 404 as i would expect.
ERROR : file.ext: Failed to copy: googleapi: Error 404: File not found:, notFound

and then i followed https://www.makeuseof.com/tag/transfer-files-google-drive-accounts/
and tried the same exact command and now it works.

rclone copy edu:source personal:dest --drive-server-side-across-configs -vv 
DEBUG : rclone: Version "v1.60.0" starting with parameters ["C:\\data\\rclone\\rclone_1.60.0.exe" "copy" "edu:source" "personal:dest" "--drive-server-side-across-configs" "-vv"]
DEBUG : Creating backend with remote "edu:source"
DEBUG : Creating backend with remote "personal:dest"
DEBUG : file.ext: md5 = f6a70a3c0f7711e6a9d7e2b539b6264f OK
INFO  : file.ext: Copied (server-side copy)
INFO  : 
Transferred:   	        2 GiB / 2 GiB, 100%, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         2.5s

Animosity022 · November 10, 2022, 10:32pm

Looks like everything works well with a nice and clean command.

Just need the OP's info and should be a nice clean solution.

fradri · November 11, 2022, 8:39am

Hello, what a nice surprise this morning to see that you two have kept going to try to help me out. This is so appreciated.

I am very happy to do some tests - I will just need some help in how to set them up/write them correctly as I'm not an expert on rclone (literally found out about this a few days ago and opened the terminal in my Macbook for the first time in my life).

This is the code we are testing is that right?

What type of info do I need to provide?
I am imagining I need to replace "source" and "dest" with my info...

Is this the path of one folder in my edu and a folder in my personal?
If I don't have any folder in my personal directory, should I create one?

Also, will this use my own internet bandwidth, and if so:

What happens if I lose connectivity?
How about if I close my laptop or if I shut it off?
I work from home: will my internet connection be affected at all?

My fear of doing this over 2+ weeks is the increased possibility of some "incidents" happening especially if my laptop needs to be operative 24/7 during that time.

Thank you both!

asdffdsa · November 11, 2022, 2:18pm

do the following.

using gdrive website, on edu:, create a folder named source
put a single small text file named file.ext in source

3a. Right-click on source and select Share
3b. type in the email address associated with the personal gdrive account
3c. match other settings and click Share
3d. the icon for source should look like this, notice the little person, that means the folder is shared.

Log into personal:. Click on the Shared with me filter on the left sidebar to quickly locate the shared folder.

image1273×359 23.2 KB
using gdrive website, on personal:, create a folder named dest
post the output of
rclone copy edu:source personal:dest --drive-server-side-across-configs -vv --dry-run

fradri · November 11, 2022, 2:42pm

Sorry already having problems at step 2

Do I need to write anything in the text file?
Also, my text edit renames it file.ext.rtf
Deleting "rtf" from the extension doesn't seem to do anything - the icon of the file appears as a "doc" file in google and not as a video. Is this going to be a problem?

I'm going to go ahead with the rest in the meantime

asdffdsa · November 11, 2022, 2:47pm

on local machine, create a text file named file.ext or file.txt,
if you want, type in some random text.
upload it to the source folder in edu

fradri · November 11, 2022, 2:51pm

Here's the result of the dry run, I wish I knew if it was positive or negative (just deleted my full name in the code)

2022/11/11 14:45:00 DEBUG : rclone: Version "v1.60.0" starting with parameters ["rclone" "copy" "edu:source" "personal:dest" "--drive-server-side-across-configs" "-vv" "--dry-run"]
2022/11/11 14:45:00 DEBUG : Creating backend with remote "edu:source"
2022/11/11 14:45:00 DEBUG : Using config file from "/Users/myfullname/.config/rclone/rclone.conf"
2022/11/11 14:45:00 DEBUG : edu: detected overridden config - adding "{Db_Y9}" suffix to name
2022/11/11 14:45:00 DEBUG : edu{Db_Y9}: Loaded invalid token from config file - ignoring
2022/11/11 14:45:00 DEBUG : Saving config "token" in section "edu" of the config file
2022/11/11 14:45:00 DEBUG : edu{Db_Y9}: Saved new token in config file
2022/11/11 14:45:00 DEBUG : Google drive root 'source': 'root_folder_id = 0AOgu6CzOixFsUk9PVA' - save this in the config to speed up startup
2022/11/11 14:45:01 DEBUG : fs cache: renaming cache item "edu:source" to be canonical "edu{Db_Y9}:source"
2022/11/11 14:45:01 DEBUG : Creating backend with remote "personal:dest"
2022/11/11 14:45:01 DEBUG : personal: detected overridden config - adding "{Db_Y9}" suffix to name
2022/11/11 14:45:01 DEBUG : personal{Db_Y9}: Loaded invalid token from config file - ignoring
2022/11/11 14:45:01 DEBUG : Saving config "token" in section "personal" of the config file
2022/11/11 14:45:01 DEBUG : personal{Db_Y9}: Saved new token in config file
2022/11/11 14:45:01 DEBUG : Google drive root 'dest': 'root_folder_id = 0AJTANAp9RP2kUk9PVA' - save this in the config to speed up startup
2022/11/11 14:45:02 DEBUG : fs cache: renaming cache item "personal:dest" to be canonical "personal{Db_Y9}:dest"
2022/11/11 14:45:02 DEBUG : Google drive root 'dest': Waiting for checks to finish
2022/11/11 14:45:02 DEBUG : Google drive root 'dest': Waiting for transfers to finish
2022/11/11 14:45:02 NOTICE: file.ext: Skipped copy as --dry-run is set (size 220)
2022/11/11 14:45:02 NOTICE: 
Transferred:   	        220 B / 220 B, 100%, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         2.1s

2022/11/11 14:45:02 DEBUG : 13 go routines active

Does this help?

I've done this with the "Text Edit" app of my Macbook but still resulted in a RTF document. I'm going to try with another software and see if I can manage to get that type of extension. In the meantime, thanks for the detailed step by step!

asdffdsa · November 11, 2022, 2:52pm

now post the output of
rclone copy edu:source personal:dest --drive-server-side-across-configs -vv

fradri · November 11, 2022, 2:54pm

Here you go -

2022/11/11 14:53:04 DEBUG : rclone: Version "v1.60.0" starting with parameters ["rclone" "copy" "edu:source" "personal:dest" "--drive-server-side-across-configs" "-vv"]
2022/11/11 14:53:04 DEBUG : Creating backend with remote "edu:source"
2022/11/11 14:53:04 DEBUG : Using config file from "/Users/myfullname/.config/rclone/rclone.conf"
2022/11/11 14:53:04 DEBUG : edu: detected overridden config - adding "{Db_Y9}" suffix to name
2022/11/11 14:53:05 DEBUG : Google drive root 'source': 'root_folder_id = 0AOgu6CzOixFsUk9PVA' - save this in the config to speed up startup
2022/11/11 14:53:05 DEBUG : fs cache: renaming cache item "edu:source" to be canonical "edu{Db_Y9}:source"
2022/11/11 14:53:05 DEBUG : Creating backend with remote "personal:dest"
2022/11/11 14:53:05 DEBUG : personal: detected overridden config - adding "{Db_Y9}" suffix to name
2022/11/11 14:53:05 DEBUG : Google drive root 'dest': 'root_folder_id = 0AJTANAp9RP2kUk9PVA' - save this in the config to speed up startup
2022/11/11 14:53:06 DEBUG : fs cache: renaming cache item "personal:dest" to be canonical "personal{Db_Y9}:dest"
2022/11/11 14:53:06 DEBUG : Google drive root 'dest': Waiting for checks to finish
2022/11/11 14:53:06 DEBUG : Google drive root 'dest': Waiting for transfers to finish
2022/11/11 14:53:08 DEBUG : file.ext: md5 = ce135150611bf9c38cc120e22eb86e28 OK
2022/11/11 14:53:08 INFO  : file.ext: Copied (server-side copy)
2022/11/11 14:53:08 INFO  : 
Transferred:   	        220 B / 220 B, 100%, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         3.7s

2022/11/11 14:53:08 DEBUG : 9 go routines active

Good news?

asdffdsa · November 11, 2022, 2:55pm

yes, looks good.

fradri · November 11, 2022, 2:55pm

Yess! That's so good to hear, made my day!

So now, what's next?

asdffdsa · November 11, 2022, 3:03pm

now,

from edu:, choose a folder, let's call it the.folder with a small amount of files in it.
move edu:the.folder into edu:source
as before run
rclone copy edu:source personal:dest --drive-server-side-across-configs -vv --dry-run
if the output looks ok, then run
rclone copy edu:source personal:dest --drive-server-side-across-configs -vv
run rclone ls personal:dest, should see the the.folder and the files inside of it.

fradri · November 11, 2022, 3:05pm

Thank you thank you thank you!!

Before I start:

Will this be dependent on my bandwidth and take 13+ days?
During the time that is computing, do I need to make sure my laptop stays on and connected to the wifi?

I'm so excited, you have no idea how much you are helping me, I was desperate.

Update: I realized we were still testing and went ahead. The small folder was successfully copied over the personal one!

asdffdsa · November 11, 2022, 3:16pm

no, as Copied (server-side copy) is working.
to transfer 2048MiB at 750MiB/day = 2.73 days

at this point, i think you are ready for the final command.

move folders from edu: into edu:source
run

rclone copy edu:source personal:dest --drive-server-side-across-configs  --drive-stop-on-upload-limit -P -v

fradri · November 11, 2022, 3:19pm

This is so exciting! Should I wait a bit before running the program or do you think Gdrive will move instantly all the folder in the "source" one?

If I open the folder on the web, they all appear inside. I also have it mirrored in my desktop and I can see the change has already happened (everything is inside the "source" folder).

Last but not least, do I need to make sure the computer stays on for these few days, right?

asdffdsa · November 11, 2022, 3:23pm

no need to wait

right

fradri · November 11, 2022, 3:25pm

Alright, here I go!

Hopefully, I'll be able to post SUCCESS in a few days

fradri · November 12, 2022, 9:01am

Hey there,

Now I have got this error message - does that mean I need to run the command again at 22:00:00 and it will keep going? Or do you think the limit resets daily (so at 00:00 every day rather than in a 24hrs timeframe)?

2022/11/11 21:59:56 Failed to copy with 180 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
MacBook-Pro-di-Francesco-3:~ myfullname$

Also right before that one there is this message (I think this is the first "info" message after 8 "context canceled" lines)

2022-11-11 21:59:56 ERROR : Fatal error received - not attempting retries
Transferred:   	  746.114 GiB / 939.536 GiB, 79%, 135.100 MiB/s, ETA 24m26s
Errors:               180 (fatal error encountered)
Checks:                15 / 15, 100%
Transferred:        42345 / 52353, 81%
Elapsed time:   6h34m40.8s
2022/11/11 21:59:56 INFO  : 
Transferred:   	  746.114 GiB / 939.536 GiB, 79%, 135.100 MiB/s, ETA 24m26s
Errors:               180 (fatal error encountered)
Checks:                15 / 15, 100%
Transferred:        42345 / 52353, 81%
Elapsed time:   6h34m40.8s

Right before the "context canceled" lines I see this:

2022-11-11 21:59:56 ERROR : Google drive root 'dest': Received upload limit error: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

I hope this helps in figuring out if all went smoothly so far (a bit worried about the 180 errors line) - let me know if I should provide any additional info.

Thank you!