Rclone keeps getting error reading destination directory: couldn't list directory: googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded

Help and Support
1

Hi all,

as title running my scheduled rclone script for medias when checking what to upload rclone keeps giving me “error reading destination directory: couldn’t list directory: googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded” with 1 or more files…

this cause a re-attempt after a while…
I have setted the number of retries to 3, now to 5…

there’s a workaround to avoid this problem?
this is not an upload limit problem (as now I making deltas over my entire collection), but when checking files on gdrive…

here’s a script example

N:\rclone\rclone.exe sync "H:\Anime" Media:"Anime"  --backup-dir=Media:"00 - Backup/Anime/%mydate%_%mytime%" --delete-during --checksum --verbose --transfers 4 --checkers 16 --bwlimit 6000 --contimeout 60s --timeout 300s --retries 5 --low-level-retries 10 --rc --stats 10s --stats-file-name-length 150 --exclude "desktop.ini" --copy-links  --log-file "r:\rclone scripts\media_%mydate%_%mytime%.log"
1 Like
2

another strange things I’m seeing is this:

the folder Anime has 7784 files and 360 folders…
if I’ve the rate limit exceeded rclone re-tries the scan but seems to re-scan all the folders, not only the file/folder in error…
I see after the second retry:
2019/02/24 11:17:31 ERROR : Attempt 2/5 succeeded
and then:
Checks: 15464 / 15464, 100%

is this normal?

this is not sane for the drive…

3

Did you make your own client ID/API key?

https://rclone.org/drive/#making-your-own-client-id

You have too many checkers as well and should turn that down to 4 if you want 4 transfers.

What’s your rclone version? You should be using the latest which is 1.46.

rclone version
rclone v1.46
- os/arch: linux/amd64
- go version: go1.11.5
1 Like
4

I didn’t know about using your own client id…

setting this up (not mentioned in documentation to refresh rclone token but is required ) the scan is very quick (quicker than before) and no error for the moment…

thx

as additional info I’m using rclone 1.46 windows 64bit

5

@ncw since we see a lot of these, might be worth making this a little more clear in both the documentation and in the “rclone config” setup to encourage people to create one and potentially walking them through it as part of the config creation itself for drive.

2 Likes
6

Wow, had no idea that the default client ID was shared between all rclone users. Awesome advice ITT, thanks all.

7

IMHO this as to place in top of drive page…
Is a life saver and free for all google accounts!

8

So change this

Storage> drive
Google Application Client Id - leave blank normally.
client_id>
Google Application Client Secret - leave blank normally.
client_secret>

To something like

Storage> drive
Google Application Client Id - setting your own is recommended.
See https://rclone.org/drive/#making-your-own-client-id for how to create your own.
If you leave this blank, it will use an internal key which is low performance.
client_id>
Google Application Client Secret - setting your own is recommended.
client_secret>

?

2 Likes
9

I think that is a great idea looks awesome!

10

Yes. Exactly. :blush: Especially since they’re not increasing your default client ID. Then at least people know from the get-go what they should do.

11

@calisro since it was your idea to you want to send a PR?

Here is the bit of code which needs changing: https://github.com/ncw/rclone/blob/master/backend/drive/drive.go#L189

12

Will do. Thanks Nick.

1 Like