Transfer 2TB of data from a Google Drive Edu account into Google One

Help and Support
81

So, it appears there has been some duplicates happening. A bunch of "NOTICE" ending in Found 2 files with duplicate names and Skipped delete as --dry-run is set (size 1.801Mi).

Sorry not 100% what to look at to see if all looks ok - it seems it has found duplicated which is what the command is intended for, so I am assuming is safe to proceed. After all it's the dest drive and worse comes to worse I can always copy again from the source.

Beforee discovering rclone, I have used the "Start Transfer" function in my edu account. It would always fail after a few days or a week and it would send me a message like this one:

Problem with your Google content transfer
There was a problem with the Google content transfer you started on Nov 4, 2022, and it didn’t finish. Try starting a new transfer, or talk to your admin. Learn moreLearn more

When checking my personal account, I would have 1.54TB (the amount I currently have). At the time, I thought that it was a Google Drive bug - after all everyone (Google customer service includeed) were telling me that it's not possible to move files out from a "workplace" account to a personal account in any way other than using "Google Takeout". But now, seeing that I'm encountering the same problem with rclone I wonder if there's a problem in my files...

82

well, that makes sense, as when you use rclone, there is an implicit --magic

and i can imagine

google api: "alert, to all troops, alert, we have an incoming @Animosity022 :bomb:
this is not a drill, repeat, this is not a drill.
double, no, triple check all api operations"

1 Like
83

let's stay focused on the dedupe, no point in discussing anything else, ok.

remember, when using --dry-run, rclone will not dedupe, will not delete. will not do anything.
so you need to dedupe the dest, by removing --dry-run

84

Sounds good!

I've run the command without the "dry run" command - the size of my personal drive hasn't changed from "1.54TB", but the command hasn't come back with any errors (just NOTICE Found 2 files with duplicate names and All duplicates removed

Should we run a check now?

rclone check edu:source personal:dest --fast-list

85

ok, good, the source is de-duped and the dest is de-duped.

now run
rclone sync edu:source personal:dest --drive-server-side-across-configs --drive-stop-on-upload-limit --fast-list -v

1 Like
86

I get this:

2022/11/14 21:10:39 INFO  : 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:       1m1.7s

2022/11/14 21:11:03 INFO  : There was nothing to transfer
2022/11/14 21:11:03 INFO  : 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Checks:             92407 / 92407, 100%
Elapsed time:      1m25.6s

Weird right? Here's a side-by-side of the storage size in each drive...

Screenshot 2022-11-14 at 21.14.22
Screenshot 2022-11-14 at 21.14.22814×388 41.2 KB

87

excellent!, the dest matches the source.

now post the output of
rclone size edu:source
and
rclone size personal:dest

1 Like
88

We got there!

2022/11/14 21:22:30 NOTICE: Google drive root 'source': Size may be underestimated due to 2 objects with unknown size
Total objects: 92.407k (92407)
Total size: 1.530 TiB (1682656067973 Byte)
Total objects with unknown size: 2 (2)

2022/11/14 21:24:09 NOTICE: Google drive root 'dest': Size may be underestimated due to 2 objects with unknown size
Total objects: 92.407k (92407)
Total size: 1.530 TiB (1682656067973 Byte)
Total objects with unknown size: 2 (2)

I wonder what the objects with unknown sizes are though and why the displayed drive size differs of 300GB on the web version of Google Drive! :thinking:

But regardless seems all is in order, right?

89

well, whatever you put into edu:source is now in personal:dest
and that was the goal.

as for whatever else is on the source gdrive, no idea.
try rclone lsd edu:

inside edu:source, do you have any google docs files, as that can confuse rclone?

This is because rclone can't find out the size of the Google docs without downloading them.

90

That's the weird thing, I only have that one "source" folder we made together

-1 2022-11-11 14:43:37        -1 source

I tried to run a rclone size edu: and I get the same results as of when I run rclone size edu:source so there must be something wrong in the web version of google drive displaying 300GB more!

2022/11/14 21:54:35 NOTICE: Google drive root '': Size may be underestimated due to 2 objects with unknown size
Total objects: 92.407k (92407)
Total size: 1.530 TiB (1682656067973 Byte)

It might be! It was an edu account originally so I might have something like that - I guess it's not important for now. :slight_smile:

Yes, and I have just you to thank me for that. I said it before but I'll say it again: thank you SO MUCH for sticking with me - it's people like you in forums like this who make internet an amazing place to be!

You have helped a complete stranger who has been stealing your time for days, and I truly appreciate it. Transferring these files for me was very important becaus that's where I have stored student, work, and life documentation and files (almost 10 years of stuff!). I had a deadline and no clue how to do the transfer between the two drives. I hope you know what a huge difference you made for me!

1 Like
91

please send that to https://rclone.org/donate/
note: i do not profit from donations.

3 Likes
92

Will do for sure!

I have chosen this as the solution for the topic - I realize we went through numerous steps (almost 100 replies!) but I think that's where we started making real progress. Let me know if you think I should choose another one instead. :slight_smile:

1 Like
93

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.