Google Drive - New API Quotas per 100 seconds

We had this issue and adjusted the min pacer sleep time to 100ms to make 10 transactions per second.

There was a post earlier and I checked my own API settings and those API quotas have been upgraded without me asking. I'd assume that's the same for majority of folks?

Any other Google Drive users can check there quota page and validate it's up to 10,000 per 100 seconds which means we should adjust the min sleep time from 100ms to 10ms since we have more capacity now.

I've noticed a significant performance improvement with making that change on my tests and will run that way for a few days to see.

I see this now in my page and I've never requested an increase.

image

7 Likes

Can confirm mine looks the same.

Not same here. I have previously requested more hence a bit different numb

Edit:
I just requested a quota increase. Up to 10000 we'll see if they grant it.

Edit2:
Did not get it. Makes sense.

Hello,

We have received your quota request for rclone-200409. Unfortunately, we are unable to grant your quota increase at this time. This is due to the project having insufficient usage history and you still have a lot of headroom.

What we highly suggest is for to make use of your current quota first and once you have used at least 80% of it, that it the best time for you to file a new request.

Additionally, you may also try implementing exponential backoff[1] and push notifications[2] to save and make use of your current quotas.

Hmm, with that data, probably don't want to change the default yet as I'm curious to see what more folks have as their quotas. I barely use my quota anyway so doesn't impact me much.

I'll continue now to make the changes via my mount command as that'll speed things up.

I have the same limits as you and never requested extra. My Client ID was created in May and only used for a daily sync.

I am currious, --tpslimit 100?

If I understand/remember how things are setup, the default tpslimit should be unlimited unless you set it.

The Google Drive has it's own pacer --drive-pacer-min-sleep which defaults to 100ms which was set to get 10 per second roughly.

I made mine 1ms to test with for now to see if I can generate any errors. I don't have a lot of use so I'm not a great test case.

1 Like

I sat mine to 10ms, and peaked at 4000 requests per 100s getting around 3 % errors during my nightly backup (this night it only verified the files, no new files).

I replied to Google and said I was hitting the rate limit. That I purposely limited my script, but that my goal is to do it faster. I'll see what they will reply, but I'm not holding my breath they will change their answer.

It's not a huge deal, but with 100ms my script usually completes around 10am (starts at 2am), so it would be great if it was done before office hours. Using 10ms (including rate limit errors) it's done around 5 am.

image
I'm seeing the new quota.

I also see the new quota... which is odd 'cos I only make around 1000 requests per week (I use grive on a one-per-hour cron schedule for 2-way sync).

1 Like

I barely use any quota at all for my setup so no idea why it would have changed, albeit it does seem like it changed for a lot of folks.

My 30 day quota usage is barely more than 12-13 per 100 seconds...

Would --drive-pacer-min-sleep 1ms have a positive effect on a rlcone copy command?

Assuming you have the quota, I could not see why not. The default limits all API hits for Google Drive to about 10 per second, which if you have more quota works, but is limiting your API hits.

I use an environment variable for my rclone.conf so you can set something like:

RCLONE_DRIVE_PACER_MIN_SLEEP=1ms in your environment so all commands use it.

can confirm this 10,000 rate limit. Have this limit since account creation. I didn't even ask about a quota increase.