Increased API errors due to increased scanning/utilization, best way to cache contents or reduce API

Help and Support
1

What is the problem you are having with rclone?

I have been trying to add more files into Google Drive, and with that, re-organize several files. Because of the increased scanning, I have run into more and more API errors on GET functions:
6 Hours

image
image944×414 41.9 KB

2 Days

image
image945×419 41.3 KB

Quota Limitations

image
image1421×501 22.4 KB

What brought this up was an increase in all media files getting stuck in 403s, not just a single file. Even after rotating out a service key, I was still getting 403s. However, this occurs for files that I have not watched in 24+ hours as well.

The only thing I can think of to do here is to potentially find a way to cache the files locally first into a local volume cache (if I am not already doing that). But I thought I already had that setup with my existing configuration.

At best, is there a way to cache reads for smaller files (like the music) and is there a way to cache the files more locally for videos? The problem I see with this is that if it attempts to cache when doing a metadata index, it will stay cached since I don't have the actual required disk space for all of these things in drive or ram space.

What is your rclone version (output from rclone version)

$ rclone version
rclone v1.52.1
- os/arch: linux/amd64
- go version: go1.14.4

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Ubuntu 20.04 LTS 64Bits

Which cloud storage system are you using? (eg Google Drive)

Google Drive, with API/Service Accounts unique for each remote.

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Paste command here

The rclone config contents with secrets removed.

Please note that the gcache remote is not in use as the plex server is not on the local server.

$ rclone config show
[ggames]
type = drive
scope = drive
token = 
team_drive = 0AMnJDL2u-UfcUk9PVA

[gdrivesoftware]
type = drive
scope = drive
token = 
team_drive = 0AGODLGuRjWlHUk9PVA

[gdrivemedia]
type = drive
scope = drive
token = 
team_drive = 0ABw3jTWf8HKKUk9PVA
client_id = 
client_secret = 

[gcache]
type = cache
remote = gdrivemedia:media
plex_url = 
plex_username = 
plex_password = 
chunk_size = 16M
chunk_total_size = 24G
info_age = 2
plex_token = 

[gmedia]
type = crypt
remote = gdrivemedia:media
filename_encryption = standard
directory_name_encryption = true
password = 
password2 = 

[gdrivemusic]
type = drive
client_id = 
client_secret = 
scope = drive
token = 
team_drive = 0AF0TKD07fcxXUk9PVA

[gdrivemusic-crypt]
type = crypt
remote = gdrivemusic:Music
filename_encryption = standard
directory_name_encryption = true
password = 
password2 = 

[gdrivegames]
type = drive
client_id = 
client_secret = 
scope = drive
token = 
team_drive = 0AMnJDL2u-UfcUk9PVA

[gdrivephotovideo]
type = drive
client_id = 
client_secret = 
scope = drive
token = 
team_drive = 0AHdD7I3gUsT1Uk9PVA

[gdriveelearning]
type = drive
client_id = 
client_secret = 
scope = drive
token = 
team_drive = 0ANAv-RkYNsrMUk9PVA

[gdriveebooks]
type = drive
client_id = 
client_secret = 
token = 
team_drive = 0APcxMq8UoECtUk9PVA

[gdriveebooks-crypt]
type = crypt
remote = gdriveebooks:/eBooks/
password = 
password2 = 

[gdrivecalibre-crypt]
type = crypt
remote = gdrivecalibre:/Calibre Library
password = 
password2 = 

[gdrivecalibre]
type = drive
client_id = 
client_secret = 
scope = drive
token = 
team_drive = 0AFmTFk6aDKpKUk9PVA

$ cat rclone-gmedia.service
[Unit]
Description=RClone Service
Wants=network-online.target
Before=docker.service
After=network-online.target

[Service]
Type=notify
Environment=RCLONE_CONFIG=/home/andrew/.config/rclone/rclone.conf
KillMode=none
RestartSec=5
ExecStart=/usr/bin/rclone mount gmedia: /mnt/rclone/gmedia \
--allow-other \
--buffer-size 256M \
--dir-cache-time 1000h \
--log-level DEBUG \
--log-file /var/log/rclone.log \
--poll-interval 15s \
--timeout 1h \
--umask 002 \
--rc \
--rc-addr 127.0.0.1:5572

ExecStop=/bin/fusermount -uz /mnt/rclone/gmedia
Restart=on-failure
User=andrew
Group=andrew

[Install]
WantedBy=multi-user.target

$ cat rclone-gmusic.service
[Unit]
Description=RClone Service
Wants=network-online.target
Before=docker.service
After=network-online.target

[Service]
Type=notify
Environment=RCLONE_CONFIG=/home/andrew/.config/rclone/rclone.conf
KillMode=none
RestartSec=5
ExecStart=/usr/bin/rclone mount gdrivemusic-crypt: /mnt/rclone/gmusic/Music \
--allow-other \
--buffer-size 64M \
--cache-dir /mnt/pool/storage/rclone/.cache/ \
--dir-cache-time 1000h \
--log-level DEBUG \
--log-file /var/log/rclone.log \
--poll-interval 15s \
--timeout 1h \
--umask 002 \
--rc \
--rc-addr 127.0.0.1:5573

ExecStop=/bin/fusermount -uz /mnt/rclone/gmusic
Restart=on-failure
User=andrew
Group=andrew

[Install]
WantedBy=multi-user.target

A log from the command with the -vv flag

2020/06/11 07:17:35 DEBUG : &{TV Shows/Doctor Who (2005)/Season 1/Doctor Who (2005) - S01E12 - Bad Wolf (1).mkv (r)}: >Read: read=0, err=open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
2020/06/11 07:17:35 DEBUG : &{TV Shows/Doctor Who (2005)/Season 1/Doctor Who (2005) - S01E12 - Bad Wolf (1).mkv (r)}: Read: len=4096, offset=0
2020/06/11 07:17:35 DEBUG : TV Shows/Doctor Who (2005)/Season 1/Doctor Who (2005) - S01E12 - Bad Wolf (1).mkv: ChunkedReader.openRange at 0 length 134217728
2020/06/11 07:17:35 DEBUG : &{TV Shows/Doctor Who (2005)/Season 1/Doctor Who (2005) - S01E12 - Bad Wolf (1).mkv (r)}: >Read: read=0, err=open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded

$ tail -n 500 -f /var/log/rclone.log | grep -A 5 -i "Starvin"
2020/06/11 07:54:05 DEBUG : TV Shows/South Park (1997)/Season 1/: Lookup: name="South Park - S01E08 - Starvin' Marvin.mkv"
2020/06/11 07:54:05 DEBUG : TV Shows/South Park (1997)/Season 1/: >Lookup: node=TV Shows/South Park (1997)/Season 1/South Park - S01E08 - Starvin' Marvin.mkv, err=<nil>
2020/06/11 07:54:05 DEBUG : TV Shows/South Park (1997)/Season 1/South Park - S01E08 - Starvin' Marvin.mkv: Attr:
2020/06/11 07:54:05 DEBUG : TV Shows/South Park (1997)/Season 1/South Park - S01E08 - Starvin' Marvin.mkv: >Attr: a=valid=1s ino=0 size=898442971 mode=-rw-rw-r--, err=<nil>
2020/06/11 07:54:05 DEBUG : TV Shows/South Park (1997)/Season 1/South Park - S01E08 - Starvin' Marvin.mkv: Open: flags=OpenReadOnly
2020/06/11 07:54:05 DEBUG : TV Shows/South Park (1997)/Season 1/South Park - S01E08 - Starvin' Marvin.mkv: Open: flags=O_RDONLY
2020/06/11 07:54:05 DEBUG : TV Shows/South Park (1997)/Season 1/South Park - S01E08 - Starvin' Marvin.mkv: >Open: fd=TV Shows/South Park (1997)/Season 1/South Park - S01E08 - Starvin' Marvin.mkv (r), err=<nil>
2020/06/11 07:54:05 DEBUG : TV Shows/South Park (1997)/Season 1/South Park - S01E08 - Starvin' Marvin.mkv: >Open: fh=&{TV Shows/South Park (1997)/Season 1/South Park - S01E08 - Starvin' Marvin.mkv (r)}, err=<nil>
2020/06/11 07:54:05 DEBUG : &{TV Shows/South Park (1997)/Season 1/South Park - S01E08 - Starvin' Marvin.mkv (r)}: Read: len=32768, offset=0
2020/06/11 07:54:05 DEBUG : TV Shows/South Park (1997)/Season 1/South Park - S01E08 - Starvin' Marvin.mkv: ChunkedReader.openRange at 0 length 134217728
2020/06/11 07:54:06 DEBUG : &{TV Shows/South Park (1997)/Season 1/South Park - S01E08 - Starvin' Marvin.mkv (r)}: >Read: read=0, err=open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
2020/06/11 07:54:06 DEBUG : &{TV Shows/South Park (1997)/Season 1/South Park - S01E08 - Starvin' Marvin.mkv (r)}: Read: len=4096, offset=0
2020/06/11 07:54:06 DEBUG : TV Shows/South Park (1997)/Season 1/South Park - S01E08 - Starvin' Marvin.mkv: ChunkedReader.openRange at 0 length 134217728
2020/06/11 07:54:06 DEBUG : TV Shows/Arrested Development (2003)/Season 5/: >ReadDirAll: item=16, err=<nil>
2020/06/11 07:54:06 DEBUG : /: Lookup: name="TV Shows"
2020/06/11 07:54:06 DEBUG : /: >Lookup: node=TV Shows/, err=<nil>
2020/06/11 07:54:06 DEBUG : TV Shows/: Attr:
2020/06/11 07:54:06 DEBUG : TV Shows/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
--
2020/06/11 07:54:06 DEBUG : &{TV Shows/South Park (1997)/Season 1/South Park - S01E08 - Starvin' Marvin.mkv (r)}: >Read: read=0, err=open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
2020/06/11 07:54:06 DEBUG : &{TV Shows/South Park (1997)/Season 1/South Park - S01E08 - Starvin' Marvin.mkv (r)}: Read: len=4096, offset=0
2020/06/11 07:54:06 DEBUG : TV Shows/South Park (1997)/Season 1/South Park - S01E08 - Starvin' Marvin.mkv: ChunkedReader.openRange at 0 length 134217728
2020/06/11 07:54:06 DEBUG : Google drive root 'media': Checking for changes on remote
2020/06/11 07:54:06 DEBUG : &{TV Shows/South Park (1997)/Season 1/South Park - S01E08 - Starvin' Marvin.mkv (r)}: >Read: read=0, err=open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
2020/06/11 07:54:06 DEBUG : TV Shows/South Park (1997)/Season 1/South Park - S01E08 - Starvin' Marvin.mkv: Attr:
2020/06/11 07:54:06 DEBUG : TV Shows/South Park (1997)/Season 1/South Park - S01E08 - Starvin' Marvin.mkv: >Attr: a=valid=1s ino=0 size=898442971 mode=-rw-rw-r--, err=<nil>
2020/06/11 07:54:06 DEBUG : TV Shows/Arrested Development (2003)/Season 1/: >ReadDirAll: item=22, err=<nil>
2020/06/11 07:54:06 DEBUG : TV Shows/Arrested Development (2003)/Season 1/: Attr:
2020/06/11 07:54:06 DEBUG : TV Shows/Arrested Development (2003)/Season 1/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2020/06/11 07:54:06 DEBUG : TV Shows/Arrested Development (2003)/Season 1/: Lookup: name="Arrested Development (2003) - S01E01 - Pilot - Bluray-1080p Remux.mkv"
2020/06/11 07:54:06 DEBUG : TV Shows/Arrested Development (2003)/Season 1/: >Lookup: node=TV Shows/Arrested Development (2003)/Season 1/Arrested Development (2003) - S01E01 - Pilot - Bluray-1080p Remux.mkv, err=<nil>
--
2020/06/11 07:54:10 DEBUG : &{TV Shows/South Park (1997)/Season 1/South Park - S01E08 - Starvin' Marvin.mkv (r)}: Flush:
2020/06/11 07:54:10 DEBUG : &{TV Shows/South Park (1997)/Season 1/South Park - S01E08 - Starvin' Marvin.mkv (r)}: >Flush: err=<nil>
2020/06/11 07:54:10 DEBUG : &{TV Shows/South Park (1997)/Season 1/South Park - S01E08 - Starvin' Marvin.mkv (r)}: Flush:
2020/06/11 07:54:10 DEBUG : &{TV Shows/South Park (1997)/Season 1/South Park - S01E08 - Starvin' Marvin.mkv (r)}: >Flush: err=<nil>
2020/06/11 07:54:10 DEBUG : &{TV Shows/South Park (1997)/Season 1/South Park - S01E08 - Starvin' Marvin.mkv (r)}: Release:
2020/06/11 07:54:10 DEBUG : &{TV Shows/South Park (1997)/Season 1/South Park - S01E08 - Starvin' Marvin.mkv (r)}: >Release: err=<nil>
2020/06/11 07:54:10 DEBUG : TV Shows/Zoids - Chaotic Century (2002)/Season 1/: >ReadDirAll: item=34, err=<nil>
2020/06/11 07:54:10 DEBUG : /: Lookup: name="TV Shows"
2020/06/11 07:54:10 DEBUG : /: >Lookup: node=TV Shows/, err=<nil>
2020/06/11 07:54:10 DEBUG : TV Shows/: Attr:
2020/06/11 07:54:10 DEBUG : TV Shows/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>

Lidarr Log (adding only to demonstrate that this is happening between different remote drives):

$ cat /mnt/pool/storage/docker/config/lidarr/logs/Lidarr.debug.txt
20-6-11 13:34:37.5|Error|AudioTag|Tag reading failed for /media/AutoLaser/01 Welcome To Jamrock (AutoLaser Bootleg Remix).mp3

[v0.7.1.1693] System.IO.IOException: Input/output error
   at System.IO.FileStream.CheckFileCall(Int64 result, Boolean ignoreNotSupported)
   at System.IO.FileStream.ReadNative(Span`1 buffer)
   at System.IO.FileStream.ReadSpan(Span`1 destination)
   at System.IO.FileStream.Read(Byte[] array, Int32 offset, Int32 count)
   at TagLib.File.ReadBlock(Int32 length)
   at TagLib.NonContainer.StartTag.ReadTagInfo(Int64& position)
   at TagLib.NonContainer.StartTag.ReadTag(Int64& start, ReadStyle style)
   at TagLib.NonContainer.StartTag.Read(ReadStyle style)
   at TagLib.NonContainer.File.Read(ReadStyle propertiesStyle)   at TagLib.File.Create(IFileAbstraction abstraction, String mimetype, ReadStyle propertiesStyle)
   at NzbDrone.Core.MediaFiles.AudioTag.Read(String path) in d:\a\1\s\src\NzbDrone.Core\MediaFiles\AudioTag.cs:line 73

20-6-11 13:34:37.6|Debug|QualityParser|Trying to parse quality for /media/AutoLaser/01 Welcome To Jamrock (AutoLaser Bootleg Remix).mp3
20-6-11 13:34:37.6|Debug|AudioTag|Unable to parse qulity from tag, Quality parsed from file path: Unknown v1, Source: Extension
20-6-11 13:34:37.6|Info|ImportDecisionMaker|Reading file 58/82600
20-6-11 13:34:37.6|Debug|AudioTag|Starting tag read for /media/AutoLaser/Falling Down (AutoLaser Bootleg Remix).mp3
20-6-11 13:34:40.3|Error|AudioTag|Tag reading failed for /media/AutoLaser/Falling Down (AutoLaser Bootleg Remix).mp3

[v0.7.1.1693] System.IO.IOException: Input/output error
   at System.IO.FileStream.CheckFileCall(Int64 result, Boolean ignoreNotSupported)
   at System.IO.FileStream.ReadNative(Span`1 buffer)
   at System.IO.FileStream.ReadSpan(Span`1 destination)
   at System.IO.FileStream.Read(Byte[] array, Int32 offset, Int32 count)
   at TagLib.File.ReadBlock(Int32 length)
   at TagLib.NonContainer.StartTag.ReadTagInfo(Int64& position)
   at TagLib.NonContainer.StartTag.ReadTag(Int64& start, ReadStyle style)
   at TagLib.NonContainer.StartTag.Read(ReadStyle style)
   at TagLib.NonContainer.File.Read(ReadStyle propertiesStyle)   at TagLib.File.Create(IFileAbstraction abstraction, String mimetype, ReadStyle propertiesStyle)
   at NzbDrone.Core.MediaFiles.AudioTag.Read(String path) in d:\a\1\s\src\NzbDrone.Core\MediaFiles\AudioTag.cs:line 73

20-6-11 13:34:40.3|Debug|QualityParser|Trying to parse quality for /media/AutoLaser/Falling Down (AutoLaser Bootleg Remix).mp3
20-6-11 13:34:40.3|Debug|AudioTag|Unable to parse qulity from tag, Quality parsed from file path: Unknown v1, Source: Extension
20-6-11 13:34:40.3|Info|ImportDecisionMaker|Reading file 59/82600
20-6-11 13:34:40.3|Debug|AudioTag|Starting tag read for /media/AutoLaser/Folivora (Welcome Home).mp3

This directly talks with rclone mount ^

1 Like
2

The cache backend is eventually going to retire so you should watch the thread:

There isn't magic really to do with download quotas. some folks have done things with rotating keys or whatnot but that's out of my scope of advice.

For that issue with 403s, you just have to wait for quota reset.

3

So to clarify, is that released in production line or beta line? Because those release versions are the same as what I am running right now?

4

Neither. You'd have to grab that beta from the link above.

@ncw is going to merge it into the mainline beta branch at "soon" and with more people testing, the quicker that happens. That linked post will have that info as well once it happens.

5

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.