Googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota

Help and Support
#1

What is the problem you are having with rclone?

I an trying to sync 2 remotes , both being shared drives . Its continuously spamming the error mentioned in question title for more than 2 hours and is still going on

What is your rclone version (output from rclone version)

rclone v1.54.0-DEV
  • os/arch: android/arm64
  • go version: go1.15.6

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Android

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone --config=$HOME/easyclone/rc.conf sync src: dst: --fast-list --drive-server-side-across-configs --drive-acknowledge-abuse --drive-keep-revision-forever --stats=10s -vvP --drive-chunk-size 128M --drive-use-trash=false --log-file=$HOME/easyclone/log.txt

The rclone config contents with secrets removed.

[src]
type = drive
scope = drive
team_drive = XXX..
service_account_file = $HOME/easyclone/accounts/1.json

[dst]
type = drive
scope = drive
root_folder_id = XXX..
service_account_file = $HOME/easyclone/accounts/2.json

A log from the command with the -vv flag

The whole log is way too large and repetitive so here's the gdrive link

Note -

  1. i am very well aware and noticed that this error is not always like fatal . I have noticed that whenever we start any rclone operation for the initial 1 or 2 mins it usually spams this error then resume normally and start listing all files but currently it's been over 2 hrs and all it is doing is spamming this continuously and it hasn't even started to list anything from my remote. . Even after hours , all there is in logs is above part from logs * many times and its still going on .

  2. It is very unlikely that my upload limit is over because i hardly even used rclone today except for checking size and syncing a already identical remote remotes once

  3. Also my service account has proper access to both the remotes no issues in that . rclone --config=$HOME/easyclone/rc.conf lsd src: && rclone --config=$HOME/easyclone/rc.conf lsd dst: properly return the folders inside the remotes

Attached below is the screenshot of the project api quota link which is being spammed

Screenshot_20210219-152016
Screenshot_20210219-1520161080×1920 139 KB

#2

Those are benign errors and means you are hitting the API too hard as you've adjusted the defaults to 8 transfers and 16 checkers and since you can only do ~10 API hit per second, your adjustments will produce errors backing off the API.

Best to use the defaults with Google Drive.

#3

Ah thanks for pointing that out . I had forgot to mention . I have already tried the sync without any checkers and transfers flag and waited for an hour but to no avail . It stil spammed these error 403 over and over abnormally for very long . In normally cases , i always see these errors get silent within 1,2 minutes . Surely something else is a miss now

#4

You'd want to include a full log.

#5

Sure , updated in post
Also since this instance of sync being logged didn't used the trackers and checkers flag used so i removed from the command as well

#6

I have no idea what changed if you edit the post so that makes it super tough.

The log file only has a clip of the repeating errors so the whole log file is really what is needed.

#7

The log which i posted in post is the whole 700kb sized log which rclone captured after running for around 25 mins before I manually cancelled it as it was spamming same thing over and over so i didn't do it for very long

#8

The log posted is a snippet / partial as the beginning parts of the command are not shown. If you can share the full log, that super helps.

#9

Updated again with the part that was missing . Sorry for confusion. The top 10,12 lines were missing from my log originally because i didn't used --log-file flag originally but captured it using tee as i like to print the debug logs in console at the same time while they are being saved .The current logs is around 7,8 minutes of rclone sync throwing the very same errors

#10

Do you have other application/programs/mounts/rclone processes using the same account? You only get 10 API hits per second.

#11

Nope , i currently don't have any other mounts or any different rclone process running anywhere

#12

You are using service accounts so not sure how they are impacted by changes made recently.

You can:

  • Test without a service account
  • Using tpslimiting and find the sweet spot of enough but not too much

Those errors just mean you are hitting the API too hard and you need back down.

#13

At this point , i am totally clueless
Rclone is throwing errors even while moving files and surprisingly gclone which is modded rclone seems to work well
I intentionally used move as to level up their differences since move doesn't have any google sided limit so there's no question of bypassing limit of any kind done by gclone which rclone can't

Here's a link to small screen recording of this comparison . I have them both execute with --dry-run having exactly same flags

#14

Not a clue what gclone is.

I'm not sure what the actual problem as you are hitting the API too hard so you'd have to back things down.

#18
#19
#20