Google Drive to Google Drive

Help and Support
1

Hey all!

I’ve been struggling with an issue for some months now with rClone when doing a Sync from One Google Drive to Another.

The size of the Sync job is about 3TB.

I’m using version 1.37 for Ubuntu

The command im using is the following:

rclone sync --verbose --transfers 16 --checkers 8 --contimeout 60s --timeout 300s --retries 3 --low-level-retries 10 --stats 5s Backup-crypt:DATA Backup1-crypt:DATA

Now I am aware that Google Drive has a 24 hour limit of ~720GB and the total job is 3TB. Typically what it would do once it reaches the 24 hour limit is that I would get an API error.

Instead I come back a few days later only to find that I have uploaded 4TB and that the transfer is still going. After watching it for a bit I also notice that the same media appears to be syncing over and over again. The other weird thing is that I am completely busting the ~720GB limit that Google is imposing and when I check the bandwidth graphs I can confirm that It’s actually uploading that much.

I should mention that I also have another data container that I synchronize once per month that is slightly bigger than the 3TB and I have no issues at all (I use the same command shown above with different path).

Has anyone come across something similar or may know what could be causing this behavior to happen?

2

The syncing multiple times is likely caused by a combination of a bug in rclone and duplicate directories.

I suggest you upgrade to rclone 1.39 or the latest beta then run rclone dedupe on your source and destination drives.

3

Hey!

Thank you so much for the information you provided.

I tried this out with dedupe rename for dedupe on both source and target and than started a new transfer.

After checking in about 20 hours later this is the stats I got. Notice the 5TB+ transfer which technically should have been stopped at ~750GB limit.

Should I try troubleshooting with a different dedupe mode?

2018/02/06 12:42:51 INFO  :
Transferred:   5913.860 GBytes (82.987 MBytes/s)
Errors:                15
Checks:              4056
Transferred:            3
Elapsed time:  20h16m12.5s
4

No dedupe rename should have done the trick.

Can you post a log somewhere? Or email it to me nick@craig-wood.com ?

5

This is interesting because when trying to transfer from a google cloud compute persistent disk to google drive I am being stopped by this 750gb limit. Hmm. I wonder if they moved GCC cloud disks to a different building than normal google drive disks. Last year this trick used to work.

6

Hey!

I did some extra troubleshooting and while logging in debug mode I found the following error

2018/02/07 11:59:12 DEBUG : pacer: Rate limited, sleeping for 1.591074439s (1 consecutive low level retries)
2018/02/07 11:59:12 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded, userRateLimitExceeded)
2018/02/07 11:59:12 DEBUG : pacer: Rate limited, sleeping for 2.472757713s (2 consecutive low level retries)
2018/02/07 11:59:12 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded, userRateLimitExceeded)
2018/02/07 11:59:12 DEBUG : pacer: Resetting sleep to minimum 10ms on success
2018/02/07 11:59:12 DEBUG : pacer: Rate limited, sleeping for 1.798169804s (1 consecutive low level retries)
2018/02/07 11:59:12 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded, userRateLimitExceeded)
2018/02/07 11:59:12 DEBUG : pacer: Rate limited, sleeping for 2.988033964s (2 consecutive low level retries)
2018/02/07 11:59:12 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded, userRateLimitExceeded)
2018/02/07 11:59:12 DEBUG : pacer: Rate limited, sleeping for 4.268626852s (3 consecutive low level retries)
2018/02/07 11:59:12 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded, userRateLimitExceeded)
2018/02/07 11:59:12 DEBUG : pacer: Resetting sleep to minimum 10ms on success

This error shows up very early in the logs however the transfer still appears to be happening (I’m seeing in vnstats im seeing an average transfer speed of 88MBytes/s)

I’ll be getting the full logs for you shortly @ncw

7

That isn’t an error - that is google doing rate limiting. As you can see it all continued as it should!