GDrive - 403 Download Quota -- Past day or two

[bad1dea (khuong)]

What is the problem you are having with rclone?

Been using Rclone / Plex / MergerFS for years now, and only starting to get 403 Errors as of.. Nov 1 - Quota Reset this morning around 2-3am EST.. i.e I could download files from my drive.. as a test.

Currently I can stream/download this file from drive.google.com but can't access it via Rclone.

Getting 403 downloadQuotaExceeded Error out of no where.

<7>DEBUG : /: Attr: 
<7>DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
<7>DEBUG : /: Lookup: name="tv"
<7>DEBUG : /: >Lookup: node=tv/, err=<nil>
<7>DEBUG : tv/: Attr: 
<7>DEBUG : tv/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
<7>DEBUG : tv/: Lookup: name="Sleepy Hollow"
<7>DEBUG : tv/: >Lookup: node=tv/Sleepy Hollow/, err=<nil>
<7>DEBUG : tv/Sleepy Hollow/: Attr: 
<7>DEBUG : tv/Sleepy Hollow/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
<7>DEBUG : tv/Sleepy Hollow/: Lookup: name="Season 1"
<7>DEBUG : tv/Sleepy Hollow/: >Lookup: node=tv/Sleepy Hollow/Season 1/, err=<nil>
<7>DEBUG : tv/Sleepy Hollow/Season 1/: Attr: 
<7>DEBUG : tv/Sleepy Hollow/Season 1/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
<7>DEBUG : tv/Sleepy Hollow/Season 1/Sleepy Hollow - S01E02 - Blood Moon Bluray-1080p.mkv: Open: flags=OpenReadOnly
<7>DEBUG : tv/Sleepy Hollow/Season 1/Sleepy Hollow - S01E02 - Blood Moon Bluray-1080p.mkv: Open: flags=O_RDONLY
<7>DEBUG : tv/Sleepy Hollow/Season 1/Sleepy Hollow - S01E02 - Blood Moon Bluray-1080p.mkv: >Open: fd=tv/Sleepy Hollow/Season 1/Sleepy Hollow - S01E02 - Blood Moon Bluray-1080p.mkv (r), err=<nil>
<7>DEBUG : tv/Sleepy Hollow/Season 1/Sleepy Hollow - S01E02 - Blood Moon Bluray-1080p.mkv: >Open: fh=&{tv/Sleepy Hollow/Season 1/Sleepy Hollow - S01E02 - Blood Moon Bluray-1080p.mkv (r)}, err=<nil>
<7>DEBUG : &{tv/Sleepy Hollow/Season 1/Sleepy Hollow - S01E02 - Blood Moon Bluray-1080p.mkv (r)}: Read: len=131072, offset=0
<7>DEBUG : tv/Sleepy Hollow/Season 1/Sleepy Hollow - S01E02 - Blood Moon Bluray-1080p.mkv: ChunkedReader.openRange at 0 length 134217728
<7>DEBUG : &{tv/Sleepy Hollow/Season 1/Sleepy Hollow - S01E02 - Blood Moon Bluray-1080p.mkv (r)}: >Read: read=0, err=open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
<7>DEBUG : &{tv/Sleepy Hollow/Season 1/Sleepy Hollow - S01E02 - Blood Moon Bluray-1080p.mkv (r)}: Read: len=4096, offset=0
<7>DEBUG : tv/Sleepy Hollow/Season 1/Sleepy Hollow - S01E02 - Blood Moon Bluray-1080p.mkv: ChunkedReader.openRange at 0 length 134217728
<7>DEBUG : &{tv/Sleepy Hollow/Season 1/Sleepy Hollow - S01E02 - Blood Moon Bluray-1080p.mkv (r)}: >Read: read=0, err=open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
<7>DEBUG : &{tv/Sleepy Hollow/Season 1/Sleepy Hollow - S01E02 - Blood Moon Bluray-1080p.mkv (r)}: Read: len=4096, offset=0
<7>DEBUG : tv/Sleepy Hollow/Season 1/Sleepy Hollow - S01E02 - Blood Moon Bluray-1080p.mkv: ChunkedReader.openRange at 0 length 134217728
<7>DEBUG : &{tv/Sleepy Hollow/Season 1/Sleepy Hollow - S01E02 - Blood Moon Bluray-1080p.mkv (r)}: >Read: read=0, err=open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
<7>DEBUG : &{tv/Sleepy Hollow/Season 1/Sleepy Hollow - S01E02 - Blood Moon Bluray-1080p.mkv (r)}: Flush: 
<7>DEBUG : &{tv/Sleepy Hollow/Season 1/Sleepy Hollow - S01E02 - Blood Moon Bluray-1080p.mkv (r)}: >Flush: err=<nil>
<7>DEBUG : &{tv/Sleepy Hollow/Season 1/Sleepy Hollow - S01E02 - Blood Moon Bluray-1080p.mkv (r)}: Flush: 
<7>DEBUG : &{tv/Sleepy Hollow/Season 1/Sleepy Hollow - S01E02 - Blood Moon Bluray-1080p.mkv (r)}: >Flush: err=<nil>
<7>DEBUG : &{tv/Sleepy Hollow/Season 1/Sleepy Hollow - S01E02 - Blood Moon Bluray-1080p.mkv (r)}: Release: 
<7>DEBUG : &{tv/Sleepy Hollow/Season 1/Sleepy Hollow - S01E02 - Blood Moon Bluray-1080p.mkv (r)}: >Release: err=<nil>

What is your rclone version (output from rclone version)

rclone v1.54.0-beta.4877.cd2c06f2a
- os/arch: linux/amd64
- go version: go1.15.3

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Ubuntu 20.04.1 LTS

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The rclone config contents with secrets removed.

[tdrive]
type = drive
client_id = xxx.apps.googleusercontent.com
client_secret = xxx-Yh
token = {"access_token":"xxx}
team_drive = XXX

[Unit]
Description=RClone tDrive Service
Wants=network-online.target
After=network-online.target

[Service]
Type=notify

ExecStart=/usr/bin/rclone mount tdrive: /mnt/tdrive \
    --config=/home/khuong/.config/rclone/rclone.conf \
    --allow-other \
    --buffer-size 256M \
    --dir-cache-time 72h \
    --drive-chunk-size 32M \
    --fast-list \
    --log-level DEBUG \
    --log-file /home/khuong/logs/tdrive.log \
    --umask 002 \
    --vfs-read-chunk-size 128M \
    --vfs-read-chunk-size-limit off \
    --vfs-cache-mode writes \
    --rc \
    --rc-addr=xxx \
    --rc-user=xxx\
    --rc-pass=xxxx\
    --user-agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36"
ExecStop=/bin/fusermount -uz /mnt/tdrive
Restart=on-failure
User=khuong
Group=khuong

[CraftyClown (Richard Lapthorn)]

Hmmmm, I was really hoping not to see a post like this as I too have been getting the same issue for the past four days. In fact It kicks in at almost exactly the same time each day (around 5pm GMT)

I was hoping it was an issue on my end as I have a fairly complicated set up with mounts on two different devices, however it seems like a strange coincidence that you have started experiencing the same thing at the same time.

I take it nothing has changed with your setup?

[bad1dea (khuong)]

No settings/configurations have been touched since May 2020.

Same applications, and usage. Really strange. Literally reset this morning, and then by around 1-2pm, it was erroring out again.

[bad1dea (khuong)]

Reset at 2am, and 403 again... just now.

[bad1dea (khuong)]

[Animosity022]

The actual API calls have little to do with it as you aren't hitting API limits.

You are hitting quota limits with download/upload which is why the API fails.

[bad1dea (khuong)]

Where would I be able to start looking?

Plex has been inactive for the past 12 hours (minus scheduled tasks) no video markers/thumbnails etc.. (again nothing has changed settings).
About 20gb uploaded since 1am. (A few TV Eps + Movies).
1 Stream on PLEX last night at 2am (me watching a tv episode).

[Animosity022]

Google doesn't provide a way to track download or upload so there isn't really a way to identify what's going on if you are hitting limits. You'd have to watch your network bandwidth or track it via other days to see what's going on and if you are consuming the download/upload cap.

If you just upgraded, you could have intro detection on or other things like that. Maybe someone synced a bunch of content.

I'd start trying to figure out how to monitor my bandwidth and look for a pattern.

[bad1dea (khuong)]

Thanks for the tips.

I normally don't monitor this server as its just Media Server - but the syncing is definitely something to look at - never been an issue before, but I tossed netdata on it, and added it to my cluster, I'll monitor it for the next few days and come back and pick your brain if I can't figure it out.

Thanks for taking a quick look at it!

[bad1dea (khuong)]

Oh for transparency, I do have intro detection on (during scheduled tasks). I heavily monitor the usage on it though.

1700 not analyzed is the same day to day for the past 3-5 months.

35220 analyzed for intros
1764 not analyzed for intros
18249 have intros