We had this issue and adjusted the min pacer sleep time to 100ms to make 10 transactions per second.
There was a post earlier and I checked my own API settings and those API quotas have been upgraded without me asking. I'd assume that's the same for majority of folks?
Any other Google Drive users can check there quota page and validate it's up to 10,000 per 100 seconds which means we should adjust the min sleep time from 100ms to 10ms since we have more capacity now.
I've noticed a significant performance improvement with making that change on my tests and will run that way for a few days to see.
I see this now in my page and I've never requested an increase.
Edit:
I just requested a quota increase. Up to 10000 we'll see if they grant it.
Edit2:
Did not get it. Makes sense.
Hello,
We have received your quota request for rclone-200409. Unfortunately, we are unable to grant your quota increase at this time. This is due to the project having insufficient usage history and you still have a lot of headroom.
What we highly suggest is for to make use of your current quota first and once you have used at least 80% of it, that it the best time for you to file a new request.
Additionally, you may also try implementing exponential backoff[1] and push notifications[2] to save and make use of your current quotas.
Hmm, with that data, probably don't want to change the default yet as I'm curious to see what more folks have as their quotas. I barely use my quota anyway so doesn't impact me much.
I'll continue now to make the changes via my mount command as that'll speed things up.
I sat mine to 10ms, and peaked at 4000 requests per 100s getting around 3 % errors during my nightly backup (this night it only verified the files, no new files).
I replied to Google and said I was hitting the rate limit. That I purposely limited my script, but that my goal is to do it faster. I'll see what they will reply, but I'm not holding my breath they will change their answer.
It's not a huge deal, but with 100ms my script usually completes around 10am (starts at 2am), so it would be great if it was done before office hours. Using 10ms (including rate limit errors) it's done around 5 am.
Assuming you have the quota, I could not see why not. The default limits all API hits for Google Drive to about 10 per second, which if you have more quota works, but is limiting your API hits.
I use an environment variable for my rclone.conf so you can set something like:
RCLONE_DRIVE_PACER_MIN_SLEEP=1ms in your environment so all commands use it.
I also had 10,000 queries until last week when you look at it, but I went back in again and saw that it had gone down to 20,000.
How can I change -drive-pacer-min-sleep and --drive-pacer-burst ?
I used this to adjust mine as when it was 10,000. You can do the math and adjust for 20,000.
# My quota per user / per 100 seconds is 10,000 requests. This can be found in your quota section.
# This changes the sleep calls to something much lower to take advantage of the API boost.
# change the min sleep from 100ms
--drive-pacer-min-sleep 10ms \
# Changing to have the ability for 1000 per second
--drive-pacer-burst 1000 \
If you get 20,000 per 100 seconds, you can estimate that you want to push 200 per second ( 20000 / 100).
1000 milliseconds in a second so 1000 / 200 would be 5ms. Basically half of what you had at 10,000 per second to double it to 20,000.
For burst, you can either hard set a max or give it some room and assume you aren't going to burst solid for a full 100 seconds. I think some errors are fine so I leave mine a bit higher.
