GDrive Upload limit not reset after 3 days

[Dazeyvide]

What is the problem you are having with rclone?

I've hit the gdrive upload limit a few days ago, and I haven't been able to upload anything for the past 3 days now:

I'm running a cronjob on 3 servers that's why I have so "many" requests. (it's the data of 4 days) Am I doing something wrong? Should I not do any upload requests for 24 hours?

Run the command 'rclone version' and share the full output of the command.

rclone v1.60.0

• os/version: debian 10.13 (64 bit)
• os/kernel: 4.19.0-22-amd64 (x86_64)
• os/type: linux
• os/arch: amd64
• go/version: go1.19.2
• go/linking: static
• go/tags: none

Which cloud storage system are you using? (eg Google Drive)

GDrive aka. Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone move "source/dir" "gdrive_encrypt:/target/dir" -P --delete-empty-src-dirs --drive-stop-on-upload-limit -vv

The rclone config contents with secrets removed.

[gdrive]
type = drive
client_id = x
client_secret = x
scope = drive
token = {"access_token":"x","token_type":"Bearer","refresh_token":"x","expiry":"x"}
team_drive = x
root_folder_id = 

[gdrive_encrypt]
type = crypt
remote = gdrive:/x
password = x
password2 = x

A log from the command with the -vv flag

2022/11/18 07:09:06 DEBUG : rclone: Version "v1.60.0" starting with parameters ["/x/bin/rclone" "move" "/x/source/dir" "gdrive_encrypt:/target/dir/" "-P" "--delete-empty-src-dirs" "--drive-stop-on-upload-limit" "-vv"]
2022/11/18 07:09:06 DEBUG : Creating backend with remote "/x/source/dir"
2022/11/18 07:09:06 DEBUG : Using config file from "/x/.config/rclone/rclone.conf"
2022/11/18 07:09:06 DEBUG : Creating backend with remote "gdrive_encrypt:/target/dir/"
2022/11/18 07:09:06 DEBUG : Creating backend with remote "gdrive:/target/cji9v7hchhXYZbtk5k8f9nmk"
2022/11/18 07:09:06 DEBUG : gdrive: detected overridden config - adding "{-LSY5}" suffix to name
2022/11/18 07:09:07 DEBUG : fs cache: renaming cache item "gdrive:/target/cji9v7hchhXYZbtk5k8f9nmk" to be canonical "gdrive{-LSY5}:data/cji9v7hchhXYZbtk5k8f9nmk"
2022/11/18 07:09:07 DEBUG : fs cache: switching user supplied name "gdrive:/target/cji9v7hchhXYZbtk5k8f9nmk" for canonical name "gdrive{-LSY5}:data/cji9v7hchhXYZbtk5k8f9nmk"
2022-11-18 07:09:08 DEBUG : Encrypted drive 'gdrive_encrypt:/target/dir/': Waiting for checks to finish
2022-11-18 07:09:08 DEBUG : Encrypted drive 'gdrive_encrypt:/target/dir/': Waiting for transfers to finish
2022-11-18 07:09:08 ERROR : Google drive root 'data/cji9v7hchhXYZbtk5k8f9nmk': Received upload limit error: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
2022-11-18 07:09:08 ERROR : inner_dir/file1: Failed to copy: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
2022-11-18 07:09:08 ERROR : inner_dir/file1: Not deleting source as copy failed: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
2022-11-18 07:09:08 ERROR : Cancelling sync due to fatal error: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded
2022-11-18 07:09:08 ERROR : inner_dir/file2: Failed to copy: Post "https://www.googleapis.com/upload/drive/v3/files?alt=json&fields=id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CexplicitlyTrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%2CshortcutDetails%2CexportLinks%2CresourceKey&supportsAllDrives=true&uploadType=resumable": context canceled
2022-11-18 07:09:08 ERROR : inner_dir/file3: Failed to copy: Post "https://www.googleapis.com/upload/drive/v3/files?alt=json&fields=id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CexplicitlyTrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%2CshortcutDetails%2CexportLinks%2CresourceKey&supportsAllDrives=true&uploadType=resumable": context canceled
2022-11-18 07:09:08 ERROR : inner_dir/file4: Failed to copy: Post "https://www.googleapis.com/upload/drive/v3/files?alt=json&fields=id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CexplicitlyTrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%2CshortcutDetails%2CexportLinks%2CresourceKey&supportsAllDrives=true&uploadType=resumable": context canceled
2022-11-18 07:09:08 ERROR : inner_dir/file3: Not deleting source as copy failed: Post "https://www.googleapis.com/upload/drive/v3/files?alt=json&fields=id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CexplicitlyTrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%2CshortcutDetails%2CexportLinks%2CresourceKey&supportsAllDrives=true&uploadType=resumable": context canceled
2022-11-18 07:09:08 ERROR : inner_dir/file2: Not deleting source as copy failed: Post "https://www.googleapis.com/upload/drive/v3/files?alt=json&fields=id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CexplicitlyTrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%2CshortcutDetails%2CexportLinks%2CresourceKey&supportsAllDrives=true&uploadType=resumable": context canceled
2022-11-18 07:09:08 ERROR : inner_dir/file4: Not deleting source as copy failed: Post "https://www.googleapis.com/upload/drive/v3/files?alt=json&fields=id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CexplicitlyTrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%2CshortcutDetails%2CexportLinks%2CresourceKey&supportsAllDrives=true&uploadType=resumable": context canceled
2022-11-18 07:09:08 ERROR : Local file system at /x/source/dir: not deleting directories as there were IO errors
2022-11-18 07:09:08 ERROR : Fatal error received - not attempting retries
Transferred:              0 B / 7.635 GiB, 0%, 0 B/s, ETA -
Errors:                 4 (fatal error encountered)
Checks:                 4 / 4, 100%
Transferred:            0 / 1, 0%
Elapsed time:         2.3s
2022/11/18 07:09:08 INFO  :
Transferred:              0 B / 7.635 GiB, 0%, 0 B/s, ETA -
Errors:                 4 (fatal error encountered)
Checks:                 4 / 4, 100%
Transferred:            0 / 1, 0%
Elapsed time:         2.3s

2022/11/18 07:09:08 DEBUG : 7 go routines active
2022/11/18 07:09:08 Failed to move with 4 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded

[prizi]

Hi, welcome! Perhaps --tpslimit might help you (see docs). You might also find some useful tips in this thread.

Unfortunately I'm not very familiar with gdrive, so hopefully someone more experienced can advise on how long you need to wait. I believe the restriction should be lifted after 24h though, so it does seem a bit strange.

There's also this recent thread, which you might like to look at. It's about a different error (429), but perhaps it's related

[Dazeyvide]

Thanks for the reply, I will try that tommorow. I've stopped all my cronjobs for now and I will check if that changes anything.

I guess it's something about my setup, because I would imagine that more users would report issues if any changes on google's side happend.

The problem with other solutions like setting a max bandwidth limit and so on is that I use it on multiple machines that upload at the same time sometimes. I could technically divide the daily bandwidth limit, but that would be 2.8MB/s which means that it would take ages to upload bigger files.

[Animosity022]

Google Drive has upload limits and it really depends on how much you upload per day and how much over you go as it's not well documented and there is no way to see it.

tpslimit has nothing do with it as you are not hitting an API limit as those as different quotas and they are 1 billion hits per day or something insane.

It just means you've tripped their upload limit and nothing to do other than waiting it out. It generally clears up after 24 hours but sometimes longer depending on whatever magical formula Google has.

[Dazeyvide]

Sadly nothing changed, I tried to migrate to Dropbox, but I've also hit the download limit after like 2TB. I guess I will just leave it for a week and try again. This is definitely weird...