GDrive Download is Slow & is Ignoring "Duplicates"

Tinqer · June 18, 2019, 10:08pm

Hello!

I just got rclone set up for the first time on my Chromebook. I've got about 2tb to download across several Google Drive accounts.

The download speed is incredibly slow. It's currently transferring 4 items at a time, and each one is under 100k/s. My connection is 100Mbps down, so I should be able to do a lot more than that? It started fast but has now slowed down significantly. I left it going overnight and only a tiny bit got done.
I am using my own API credentials & the latest version of rclone.

rclone copy GoogleDriveT: /mnt/chromeos/removable/TOSHIBA/Drive --log-file=/home/user/rclone/log.log

Less pressing issue is that it is ignoring some files, saying "Duplicate object found in source". I love that it is trying to help me get rid of duplicates, but these files are actually different (diff date & file size) -- they just share the same filename. Is there an easy way to make it grab both versions?

2019/06/18 23:21:08 NOTICE: Personal/Anniversaries.docx: Duplicate object found in source - ignoring
2019/06/18 23:21:08 NOTICE: Personal/Anniversaries.docx: Duplicate object found in source - ignoring
2019/06/18 23:21:08 NOTICE: Personal/Anniversaries.docx: Duplicate object found in source - ignoring
2019/06/18 23:21:08 NOTICE: Personal/Anniversaries.docx: Duplicate object found in source - ignoring
2019/06/18 23:21:08 NOTICE: Personal/Anniversaries.docx: Duplicate object found in source - ignoring

Simple advice appreciated -- I am new to a lot of this.

Thank you!

Animosity022 · June 18, 2019, 11:04pm

Can you run the same command with "-vv" and share the log?

You can use --fast-list with Google Drive as well.

Tinqer · June 18, 2019, 11:38pm

At the moment the log has a bunch of personal info, is it ok if I pick out these generic bits? This is the stuff I've found that isn't about specific files:

2019/06/19 08:27:19 DEBUG : rclone: Version "v1.48.0" starting with parameters ["rclone" "copy" "-vv" "GoogleDriveT:" "/mnt/chromeos/removable/TOSHIBA/Drive" "--log-file=/home/user/rclone/19june.log"]
2019/06/19 08:27:19 DEBUG : Using config file from "/home/user/.config/rclone/rclone.conf"
2019/06/19 08:27:19 DEBUG : GoogleDriveT: Loaded invalid token from config file - ignoring
2019/06/19 08:27:20 DEBUG : GoogleDriveT: Saved new token in config file
2019/06/19 08:27:22 NOTICE: VW_5rm.png: Duplicate object found in source - ignoring

2019/06/19 08:27:27 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=537686328147, userRateLimitExceeded)
2019/06/19 08:27:27 DEBUG : pacer: Rate limited, increasing sleep to 1.507760161s
2019/06/19 08:27:27 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=537686328147, userRateLimitExceeded)
2019/06/19 08:27:27 DEBUG : pacer: Rate limited, increasing sleep to 2.93302406s
2019/06/19 08:27:27 DEBUG : pacer: Reducing sleep to 0s

2019/06/19 08:27:39 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=537686328147, userRateLimitExceeded)
2019/06/19 08:27:39 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=537686328147, userRateLimitExceeded)
2019/06/19 08:27:39 DEBUG : pacer: Rate limited, increasing sleep to 1.015520031s
2019/06/19 08:27:39 DEBUG : pacer: Rate limited, increasing sleep to 2.748344413s
2019/06/19 08:27:39 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=537686328147, userRateLimitExceeded)
2019/06/19 08:27:39 DEBUG : pacer: Rate limited, increasing sleep to 4.164140749s
2019/06/19 08:27:39 DEBUG : pacer: Reducing sleep to 0s
2019/06/19 08:27:39 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=537686328147, userRateLimitExceeded)
2019/06/19 08:27:39 DEBUG : pacer: Rate limited, increasing sleep to 1.168270882s
2019/06/19 08:27:39 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=537686328147, userRateLimitExceeded)
2019/06/19 08:27:39 DEBUG : pacer: Rate limited, increasing sleep to 2.778007112s
2019/06/19 08:27:39 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=537686328147, userRateLimitExceeded)
2019/06/19 08:27:39 DEBUG : pacer: Rate limited, increasing sleep to 4.455075339s
2019/06/19 08:27:39 DEBUG : pacer: Reducing sleep to 0s

Seems to say stuff like that regularly in the log.

Transferred: 1.324G / 440.744 GBytes, 0%, 746.161 kBytes/s, ETA 171h31m55s
Errors: 6 (retrying may help)
Checks: 383 / 383, 100%
Transferred: 513 / 10525, 5%
Elapsed time: 31m0s

If you need a whole log to diagnose it better let me know.

Animosity022 · June 18, 2019, 11:41pm

You don't seem to be using your own API key, Are you seeing hits in the console for it?

Tinqer · June 18, 2019, 11:44pm

Hmm, I believe so. Does this look right?

Just realised it's saying invalid token at the top of the log (updated previous post). I'll see if I can re-enter the API key etc.

Animosity022 · June 19, 2019, 12:06am

You should also run rlcone dedupe on your remote.

It's interactive by default and clean up the duplicates.

Tinqer · June 19, 2019, 12:14am

Thanks!

No luck yet on the speed unfortunately. I ensured the remote has my API info and ran it again. Starts like this:

2019/06/19 09:22:47 DEBUG : rclone: Version "v1.48.0" starting with parameters ["rclone" "copy" "-vv" "GoogleDriveT:" "/mnt/chromeos/removable/TOSHIBA/Drive" "--fast-list" "--log-file=/home/user/rclone/18juneB.log"]
2019/06/19 09:22:47 DEBUG : Using config file from "/home/thoroughmas/.config/rclone/rclone.conf"
2019/06/19 09:22:52 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=537686328147, userRateLimitExceeded)
2019/06/19 09:22:52 DEBUG : pacer: Rate limited, increasing sleep to 1.945210361s
2019/06/19 09:22:52 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=537686328147, userRateLimitExceeded)
2019/06/19 09:22:52 DEBUG : pacer: Rate limited, increasing sleep to 2.250035109s
2019/06/19 09:22:52 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=537686328147, userRateLimitExceeded)
2019/06/19 09:22:52 DEBUG : pacer: Rate limited, increasing sleep to 4.87679124s
2019/06/19 09:22:52 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=537686328147, userRateLimitExceeded)
2019/06/19 09:22:52 DEBUG : pacer: Rate limited, increasing sleep to 8.722999592s
2019/06/19 09:22:52 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=537686328147, userRateLimitExceeded)
2019/06/19 09:22:52 DEBUG : pacer: Rate limited, increasing sleep to 16.433233347s
2019/06/19 09:22:52 DEBUG : pacer: Reducing sleep to 0s

Then it gets to work, again slowly. Transfer details:

2019/06/19 09:39:47 INFO :
Transferred: 382.711M / 408.439 GBytes, 0%, 384.188 kBytes/s, ETA 309h22m24s
Errors: 29 (retrying may help)
Checks: 135 / 752, 18%
Transferred: 169 / 10189, 2%
Elapsed time: 17m0s

Animosity022 · June 19, 2019, 12:16am

Doesn't seem like the key is picking up with those rate limits.

What does your rclone.conf look like?

-- was asking without the passwords/keys/etc.

It does seem your API is there as the token looks good.

Run the dedupe. Are you trying to do a lot of little files?

Tinqer · June 19, 2019, 12:34am

Oops, sorry, thanks for deleting.

Ok I'll run the dedupe and keep trying. It is a big ~500gb Drive with many kinds of little files, yes.

Animosity022 · June 19, 2019, 12:42am

Little files are particularly bad on GD as you can only do ~8-10 transactions per second.

Tinqer · June 19, 2019, 1:14am

When I hit larger files it's still bafflingly slow.

I'll be happy to try any other ideas, otherwise might soon have to look for an alternate method of doing this.

Animosity022 · June 19, 2019, 5:24pm

I'm not sure as I can run the same tests and max out my gigabit link.

I did a pretty basic test with just 4 files copying.

Tinqer · June 19, 2019, 10:27pm

Hmm, yeah pretty frustrating. I tried from another computer (Macbook) and it was the same. Thanks heaps of helping though Animosity.

Animosity022 · June 19, 2019, 11:27pm

Did you dedupe it?

system · September 17, 2019, 11:27pm

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.