Google drive upload limit error

What is the problem you are having with rclone?

I am trying to copy files on Google Drive. I consistently receive the following error message:

Received upload limit error: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

I am trying to copy files from a directory on google drive to another directory on google drive. I have enough space on google drive to copy the files.

The copying routinely fails after some time with this error message. Typically after 18 hours or so I can resume the copying.

The message appears after transferring around ~150 GB, which is much less than the 750 GB mentioned in the google drive documentation.

I would like to know which options, if any, in rclone, would allow me to copy these files (even slowly) without getting this error. I don't need to copy the files quickly. I would like a command that would do the job, even slowly, but without stopping after 1 or 2 hours with this error message. I need to copy a lot of files. The current process is very tedious since I need to restart the copying process every day.

Is it a matter of reducing --bwlimit? I am using --bwlimit 8M right now. I thought this was sufficient to stay below google drive's quota. See below for details.

Thank you for your help.

Run the command 'rclone version' and share the full output of the command.

rclone --version
rclone v1.65.2
- os/version: ubuntu 16.04 (64 bit)
- os/kernel: 4.15.0-142-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.21.6
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone -v --tpslimit 1 --bwlimit 8M --drive-pacer-min-sleep 1000ms --drive-pacer-burst 1 --checkers 2 --drive-stop-on-upload-limit --log-file=log.txt sync gdrive:src_dir gdrive:dest_dir

Could you please suggest what options might be appropriate to stay below google drive's quotas?

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[gdrive]
type = drive
scope = drive
token = XXX
team_drive = 

A log from the command that you were trying to run with the -vv flag

2024/02/09 15:16:39 ERROR : Google drive root 'XXX': Received upload limit error: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
2024/02/09 15:16:39 ERROR : XXX: Failed to copy: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
2024/02/09 15:16:39 ERROR : Cancelling sync due to fatal error: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
2024/02/09 15:16:39 ERROR : XXX: Failed to copy: couldn't list directory: context canceled
2024/02/09 15:16:39 ERROR : XXX: Failed to copy: context canceled
2024/02/09 15:16:39 ERROR : XXX: Failed to copy: context canceled
2024/02/09 15:16:39 ERROR : Google drive root 'XXX': not deleting files as there were IO errors
2024/02/09 15:16:39 ERROR : Google drive root 'XXX': not deleting directories as there were IO errors
2024/02/09 15:16:39 ERROR : Fatal error received - not attempting retries
2024/02/09 15:16:39 INFO  :
Transferred:      107.417 GiB / 326.647 GiB, 33%, 0 B/s, ETA -
Errors:                 4 (fatal error encountered)
Transferred:          220 / 682, 32%
Server Side Copies:   220 @ 107.417 GiB
Elapsed time:      8m38.5s

I redacted some of the information with XXX. Note that I need to copy many directories. This was a command that attempted to sync only one of these directories.

What you are missing is you own client_id/secret. As by default you share one with many other rclone users you might hit various limits (it is not only daily upload but also number of transactions per time period) at unpredictable manner.

Make you own and recreate your gdrive remote providing your personalised parameters. Here also my simplified guide how to do it.

[gdrive]
type = drive
client_id = XXX
client_secret = XXX
scope = drive
token = XXX
team_drive = 
1 Like

This worked. Thank you for the detailed explanations. That was very helpful.

1 Like

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.