Google Drive - Transfers Whole File Before Rate Limit Error

Help and Support
1

What is the problem you are having with rclone?

RClone (or Google Drive) allow the whole transfer to complete BEFORE throwing the "googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded" error. So if I transfer a 100GB file, it will spend a long time transferring the file before it "completes" at which point it gives the error.

If I'm at quota, why does it allow the transfer to start at all? (and Google states if you're not at quota, but the file you're uploading would push you over quota, then that file would still be allowed).

This is obviously a huge time sync so I'm looking for some way around the issue.

What is your rclone version (output from rclone version)

rclone v1.56.2

  • os/version: arch (64 bit)
  • os/kernel: 5.14.9-arch2-1 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.17.1
  • go/linking: dynamic
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone move --progress --fast-list --drive-chunk-size 256M --ignore-existing --drive-stop-on-upload-limit --drive-stop-on-download-limit FileImTransferring GDrive:/Archive/

The rclone config contents with secrets removed.

Only non-secret lines are:

scope = drive
use_trash = false
chunk_size = 256Mi
acknowledge_abuse = true

A log from the command with the -vv flag

I can't run this again right now due to quotas and bandwidth issues, but the last error was:

2021-10-25 09:55:46 ERROR : Google drive root 'Archive': Received upload limit error: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
2021-10-25 09:55:46 ERROR : FileToTransfer: Failed to copy: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
2021-10-25 09:55:46 ERROR : FileToTransfer: Not deleting source as copy failed: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
2021-10-25 09:55:46 ERROR : Fatal error received - not attempting retries
Transferred:       37.482Gi / 37.482 GiByte, 100%, 56.111 MiByte/s, ETA 0s
Errors:                 1 (fatal error encountered)
Checks:                 1 / 1, 100%
Elapsed time:     10m31.8s
2021/10/25 09:55:46 Failed to move: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

As you can see, it transferred the whole file before reporting the user rate limit.

2

There's no magic unfortunately as that's how Google Drive quotas work.

It doesn't know the file is too big until it finishes as that is when the limit is flagged as the quota is not quite a hard thing but depends on how much you left.

You can see it more specifically in the debug log if you run it with -vv and share the full output.

3

Google servers won't let you or rclone know in advance how much quota is left.
So not much we can do here until Google introduces a specific API request.

4

Hi @Enverex , @Animosity022 and @ivandeex

I have a few curious slightly off-topic questions (while trying to improve the pacing in situations with throttling and rate limiting):

Do you know whether the “403/User rate limited” (upload quota used) is a hard stop from Google or would it be possible to continue the upload in SLOOOOW motion? (until complete or quota replenised)

What happens if you try uploading a file through the Google Drive web interface just after rclone receives this error? Will it terminate in a similar way?

5

It's a hard stop. The quota is published at 750GB upload but the way it resets / exactly how is not. Most folks thinks it's some sort of rolling limit and it gets wonky if you upload say a 1TB file, that throws you over the 750GB but no one exactly knows how long. Most say it resets some of that near 00:00 UTC.

Say you are hit something like 740GB and you upload a few small 1GB as that would work, but if you did a 15GB file, it would fail.

Quotas are API only so not related to the web interface. To the best of my knowledge, you can upload as much as you want through the web interface but you can't use rclone crypt that route which is how I upload everything.

1 Like
6

That makes sense, but the Google documentation explicitly says that transactions/uploads in process will complete if they are the specific upload to push it over the quota. Is this no-longer true, or does RClone have issues for some other reason?

1 Like
7

It doesn't know you are over quota until the upload is done though as there's no API to expose quota so the error and behavior is exactly what I would expect.

8

I am curious, can I have a link to this?

9

Sure:

Upload files to Google Drive - Google Workspace Admin Help.

10

Thought I'd chime in here, because of my years of experience of uploading the maximum allowed each day, legitimately.

For a while, I wasn't sure how the Google reset works. I believed it was random and rolling, but it's not. The reset is triggered by the first upload of the 24 hour period. So, if you start uploading at 9 AM on day 1, and you upload 750GB, you'll have to wait until 9 AM on day 2. If you happen to start your first upload of day 2 later than that, then that time + 24 hours will be the new reset time, and so on. I have closely observed this behavior over the past year, and it's been consistent throughout.

As far as going over the limit, up until about a year and half ago, there was a way to go significantly beyond the 750GB. You could, for example, upload just below that, then stop the job, and start a new one with as many transfers as you needed, and they would all still finish. Google put a stop to that a while ago, though. Now, only one single transfer will go beyond the 750GB limit. Again, this is my personal observation.

If anyone's experience is vastly different, please let us know.

1 Like
11

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.