Keep getting api bans

FER1 · May 9, 2020, 5:10pm

What is the problem you are having with rclone?

In the last week I keep getting API bans. I've changed my mount command a bit but can't seem to fix it. Would love to get some feedback

I tried to increase --vfs-read-chunk-size and introduce tpslimit

What is your rclone version (output from `rclone version`)

rclone v1.49.2

os/arch: linux/arm64
go version: go1.12.9

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Debian 9 arm64

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

Last command tried:
ExecStart=/usr/bin/rclone mount GDrive:NAS /home/rock64/GDrive
--allow-other
--umask 000
--dir-perms 777
--file-perms 777
--vfs-cache-mode writes
--vfs-read-chunk-size 512M
--dir-cache-time 96h
--vfs-cache-max-size 3G
--buffer-size 0M
--tpslimit 5
--log-file /logs/mount.log
--log-level INFO

My original settings from:
ExecStart=/usr/bin/rclone mount GDrive:NAS /home/rock64/GDrive
--allow-other
--umask 000
--dir-perms 777
--file-perms 777
--vfs-cache-mode writes
--vfs-read-chunk-size 256M
--dir-cache-time 96h
--vfs-cache-max-size 2G
--buffer-size 0M
--log-file /logs/mount.log
--log-level INFO

A log from the command with the `-vv` flag (eg output from `rclone -vv copy /tmp remote:tmp`)

2020/05/09 15:46:48 ERROR : Downloads/File3/BDMV/STREAM/00008.m2ts: ReadFileHandle.Read error: low level retry 1/10: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
2020/05/09 15:46:48 ERROR : Downloads/File3/BDMV/STREAM/00008.m2ts: ReadFileHandle.Read error: low level retry 2/10: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
2020/05/09 15:46:49 ERROR : Downloads/File3/BDMV/STREAM/00008.m2ts: ReadFileHandle.Read error: low level retry 3/10: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
2020/05/09 15:46:49 ERROR : Downloads/File3/BDMV/STREAM/00008.m2ts: ReadFileHandle.Read error: low level retry 4/10: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
2020/05/09 15:46:49 ERROR : Downloads/File3/BDMV/STREAM/00008.m2ts: ReadFileHandle.Read error: low level retry 5/10: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
2020/05/09 15:46:50 ERROR : Downloads/File3/BDMV/STREAM/00008.m2ts: ReadFileHandle.Read error: low level retry 6/10: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
2020/05/09 15:46:50 ERROR : Downloads/File3/BDMV/STREAM/00008.m2ts: ReadFileHandle.Read error: low level retry 7/10: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
2020/05/09 15:46:50 ERROR : Downloads/File3/BDMV/STREAM/00008.m2ts: ReadFileHandle.Read error: low level retry 8/10: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
2020/05/09 15:46:51 ERROR : Downloads/File3/BDMV/STREAM/00008.m2ts: ReadFileHandle.Read error: low level retry 9/10: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
2020/05/09 15:46:51 ERROR : Downloads/File3/BDMV/STREAM/00008.m2ts: ReadFileHandle.Read error: low level retry 10/10: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
2020/05/09 15:46:51 ERROR : Downloads/File3/BDMV/STREAM/00008.m2ts: ReadFileHandle.Read error: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
2020/05/09 15:46:51 ERROR : Downloads/File3/BDMV/STREAM/00008.m2ts: ReadFileHandle.Read error: low level retry 1/10: file already closed
2020/05/09 15:46:52 ERROR : Downloads/File3/BDMV/STREAM/00008.m2ts: ReadFileHandle.Read error: low level retry 2/10: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
2020/05/09 15:46:52 ERROR : Downloads/File3/BDMV/STREAM/00008.m2ts: ReadFileHandle.Read error: low level retry 3/10: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
2020/05/09 15:46:52 ERROR : Downloads/File3/BDMV/STREAM/00008.m2ts: ReadFileHandle.Read error: low level retry 4/10: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
2020/05/09 15:46:52 ERROR : Downloads/File3/BDMV/STREAM/00008.m2ts: ReadFileHandle.Read error: low level retry 5/10: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
2020/05/09 15:46:53 ERROR : Downloads/File3/BDMV/STREAM/00008.m2ts: ReadFileHandle.Read error: low level retry 6/10: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
2020/05/09 15:46:53 ERROR : Downloads/File3/BDMV/STREAM/00008.m2ts: ReadFileHandle.Read error: low level retry 7/10: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
2020/05/09 15:46:54 ERROR : Downloads/File3/BDMV/STREAM/00008.m2ts: ReadFileHandle.Read error: low level retry 8/10: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
2020/05/09 15:46:54 ERROR : Downloads/File3/BDMV/STREAM/00008.m2ts: ReadFileHandle.Read error: low level retry 9/10: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
2020/05/09 15:46:54 ERROR : Downloads/File3/BDMV/STREAM/00008.m2ts: ReadFileHandle.Read error: low level retry 10/10: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
2020/05/09 15:46:54 ERROR : Downloads/File3/BDMV/STREAM/00008.m2ts: ReadFileHandle.Read error: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
2020/05/09 15:46:55 ERROR : Downloads/File3/BDMV/STREAM/00008.m2ts: ReadFileHandle.Release error: file already closed
2020/05/09 16:30:54 INFO : Starting HTTP transaction limiter: max 5 transactions/s with burst 1

This is my api usage:

asdffdsa · May 9, 2020, 5:22pm

hello and welcome to the forum,

you are using an old version of rclone, you should update to v1.51.0.

"The download quota for this file has been exceeded., downloadQuotaExceeded"
that means you have downloaded the most gdive will allow in 24 hours, approx 750GB.
so no rclone setting will help with that.
you just have to wait.

are you using your own client id, or the shared client id that is shared with all rclone users?
https://rclone.org/drive/#making-your-own-client-id

calisro · May 9, 2020, 5:25pm

that isn't an API BAN. That is a download quota issue. I'm guessing that since your buffer is zero and your chunksize is large that your streaming product is trying to read small chunks and its downloading the large chunk to do each little chunk then throwing it away so its attempting to download all those big chunks over and over again. I'd try to reduce your chunk size. (perhaps just try the defaults)

FER1 · May 9, 2020, 5:52pm

isn't the download limit like 10TB? The upload is 750

FER1 · May 9, 2020, 5:52pm

I'll try that and will let you know if it works

asdffdsa · May 9, 2020, 5:53pm

yes, my bad. thanks

Animosity022 · May 9, 2020, 9:28pm

There is another unpublished quota that has to do with downloads per file that you are hitting.

If you check your GSuite audit log, you'd want to isolate that file and see how many times you've downloaded and check a 24 hour period.

I'd dump it into Excel and check for that specific file via a filter and see what the counts are.

Each range request shows up as a download and Google has some formula they use to decide how many audits and downloads are allowed.

Shared files seem lower. Team Drives might be lower. I personally use a regular Drive and have tested with 2500 hits on a file and didn't have the issue. I'm sure there is a number, just not sure since they don't publish it.

FER1 · May 10, 2020, 7:23am

I use a regular Drive and I got 18549. Any idea if I can reduce the hits in any way? I tried both tpslimit and increasing the vfs chunk size but nothing. Everything worked great in the past months, no idea what is causing the issue

Animosity022 · May 10, 2020, 10:49am

You got 18k hits for a single file? That seems quite a bit excessive.

You'd want to check what changed as in a new player, new thing, new something that happened as you can't tune your way out of that.

FER1 · May 10, 2020, 12:21pm

This seems to have solved the issue. Thanks a lot

system · May 13, 2020, 12:21pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.