So slow - googleapi: Error 403: Rate Limit Exceeded

Hi Folks,

I’m setting up a script that uses rclone to sync to Google Team Drive for around 450 users via our Google Apps for Business package. I’m having great difficulty with the speed of just checking if any files need syncing.

My test folder has 12.5 GB split across 217 files. All are currently up to date, so no file transfer needed. However, running sync still takes between 2 and 4.5 minutes, averaging around 2.5 minutes.

I also have an application called Syncovery installed. By comparison, that program takes only about 4 seconds to determine no files need to be transferred.

I’ve created and linked my own Google API client_id to my rclone configuration, but that doesn’t appear to have made any noticeable difference.

My sync command is simply:
rclone.exe --config c:\SYSTEM.FSO\config\rclone\rclone.conf --stats 20s --stats-log-level INFO sync -vv --delete-before Field_Laptop_Files:Folder1/Folder2 C:\Folder1\Folder2

The debug output seems to suggest constant “Error 403: Rate Limit Exceeded” right from the beginning, and it’s a good minute before the list of files being checked starts to show in the output.

Can anyone suggest anything I can do to speed things up, the difference in speed between these 2 apps is huge?

Thanks,
Samo

The latest beta I believe had some fixes for this and you could try that along with --fast-list

Are you able to see what your error rates look like in the API console? Depending on how many other users you have hitting it, you might want to throttle rclone a bit.

I limit my own quite a bit as I generally just move a few big files:

# Move older local files to the cloud
/usr/bin/rclone move /data/local/ gcrypt: --checkers 3 --fast-list --log-file /home/felix/logs/upload.log -v --tpslimit 3 --transfers 3 --exclude-from /home/felix/scripts/excludes --delete-empty-src-dirs

Depending on how much capacity you have, you could always go much higher. You effectively 10 transactions per second for 100 seconds as that hits the default quotas for your API key.

1 Like

Thanks so much for your help. The beta version didn’t seem to make much difference, but your additional parameter have bought the sync time down from 150s to 14.5s!! My Drive API was showing around 20% 403 response codes, and between 10-25% errors, it’s now showing 0% for both.

Yeah, the challenge with so many backends is that the defaults are not always perfect and some tweaking is needed. I use GD so I’m pretty familiar with using that item.

Do you think we should put some more docs in the drive page? Drive is the most popular backend so some more tweaking tips would probably be a good idea!

Definitely. I’ll see if I can write up some notes.

2 Likes

This article also discuss the chunk size … is this something that should be tuned with google as well?

https://www.chpc.utah.edu/documentation/software/rclone.php

That’s a really cool article. I’ll definitely give that options a test!