Plans to resolve Google Drive 403?

aj1252 · July 12, 2017, 8:08pm

@ncw Is there any plan to resolve the Google Drive 403?

I would settle for just something as simple as parameter to set a single threaded mount with an artificial delay between the requests even with the reduced performance. Just give us something please. There are other solutions out there but yours are the only production ready and professionally managed I’ve seen.

calisro · July 12, 2017, 11:15pm

Use your own API and keep the transfers checkers low. 403 isn’t really something that needs fixing… It’s just a rate limit by Google …

What problem is this causing you?

ncw · July 13, 2017, 3:45am

That is an interesting idea. There is a similar idea here https://github.com/ncw/rclone/issues/485 about limiting the number of files per second. However it would be relatively easy to add a limit of API transactions per second.

Let’s say I implemented a --tps-limit command which you would give a maximum number of transactions per second you wanted as a floating point number. So you could say --tps-limit 10 or --tps-limit 0.5 for 1 every 2 seconds.

Do you think that would work?

Not sure I like the name --tps-limit though… Probably --http-tps-limit would be more accurate.

ncw · July 13, 2017, 4:20am

I did a quick experiment

https://pub.rclone.org/v1.36-251-gdac56457-tps-limitβ/

This add this flag - have a go with it and see if it make a difference.

  --min-transaction-interval duration   Minimum time between HTTP transactions. (default 1ns)

aj1252 · July 13, 2017, 6:31am

It is not the Google API, but a hidden limit on Drive itself, which is the issue. The normal API limit rclone is already doing a good job throttling.

aj1252 · July 13, 2017, 6:34am

Awesome! I’ll test it shortly.

aj1252 · July 13, 2017, 8:58am

30 minutes running plex scan + playing a movie towards 1s restriction.
Watching the log looks like it obeying the 1s restriction.
No bans so far… Great.

Uping the ante, adding Sonarr and Radarr as well… Crosses fingers…

aj1252 · July 13, 2017, 9:17am

Its hard for me to see, since I don’t know what’s internal logging and what’s requests. However, running Plex scan, playing a movie, Sonarr scan and Radarr scan simultaneous. It looks to me that each is only obeying the limit for themselves. ie parallel. Here is 3 seconds from the log which should only have 3 requests:

2017/07/13 11:07:40 DEBUG : TV/: Attr:
2017/07/13 11:07:40 DEBUG : TV/: >Attr: attr=valid=1m0s ino=0 size=0 mode=drwxrwxr-x, err=
2017/07/13 11:07:40 DEBUG : TV/: ReadDirAll:
2017/07/13 11:07:40 DEBUG : TV/: >ReadDirAll: item=1772, err=
2017/07/13 11:07:40 DEBUG : TV/: Lookup: name=“A to Z”
2017/07/13 11:07:40 DEBUG : TV/: >Lookup: node=, err=no such file or directory
2017/07/13 11:07:40 DEBUG : TV/: Lookup: name=“A to Z”
2017/07/13 11:07:40 DEBUG : TV/: >Lookup: node=, err=no such file or directory
2017/07/13 11:07:40 DEBUG : TV/: Lookup: name=“A to Z”
2017/07/13 11:07:40 DEBUG : TV/: >Lookup: node=, err=no such file or directory
2017/07/13 11:07:40 DEBUG : TV/: Lookup: name=“A to Z”
2017/07/13 11:07:40 DEBUG : TV/: >Lookup: node=, err=no such file or directory
2017/07/13 11:07:40 DEBUG : TV/Adventure Time/Adventure Time - S08E09 - Islands, Part 3- Mysterious Island [WEBDL-1080p x265 AAC [EN] UTR].mkv ®: >Read: read=131072, err=
2017/07/13 11:07:40 DEBUG : TV/Adventure Time/Adventure Time - S08E09 - Islands, Part 3- Mysterious Island [WEBDL-1080p x265 AAC [EN] UTR].mkv ®: Read: len=69632, offset=97538048
2017/07/13 11:07:40 DEBUG : TV/: Lookup: name=“A to Z”
2017/07/13 11:07:40 DEBUG : TV/: >Lookup: node=, err=no such file or directory
2017/07/13 11:07:40 DEBUG : TV/: Lookup: name=“A to Z”
2017/07/13 11:07:40 DEBUG : TV/: >Lookup: node=, err=no such file or directory
2017/07/13 11:07:40 DEBUG : TV/: Attr:
2017/07/13 11:07:40 DEBUG : TV/: >Attr: attr=valid=1m0s ino=0 size=0 mode=drwxrwxr-x, err=
2017/07/13 11:07:40 DEBUG : TV/: ReadDirAll:
2017/07/13 11:07:40 DEBUG : TV/: >ReadDirAll: item=1772, err=
2017/07/13 11:07:40 DEBUG : TV/: Lookup: name=“The 9th Life of Louis Drax (2016)”
2017/07/13 11:07:40 DEBUG : TV/: >Lookup: node=, err=no such file or directory
2017/07/13 11:07:40 DEBUG : TV/: Lookup: name=“The 9th Life of Louis Drax (2016)”
2017/07/13 11:07:40 DEBUG : TV/: >Lookup: node=, err=no such file or directory
2017/07/13 11:07:40 DEBUG : TV/Adventure Time/Adventure Time - S08E09 - Islands, Part 3- Mysterious Island [WEBDL-1080p x265 AAC [EN] UTR].mkv: ReadFileHandle.seek from 131072 to 97538048 (io.Seeker)
2017/07/13 11:07:40 DEBUG : TV/: Attr:
2017/07/13 11:07:40 DEBUG : TV/: >Attr: attr=valid=1m0s ino=0 size=0 mode=drwxrwxr-x, err=
2017/07/13 11:07:40 DEBUG : TV/: ReadDirAll:
2017/07/13 11:07:40 DEBUG : TV/: >ReadDirAll: item=1772, err=
2017/07/13 11:07:40 DEBUG : TV/Abandoned Engineering/: ReadDirAll:
2017/07/13 11:07:40 DEBUG : TV/Abandoned Engineering: Re-reading directory (6m20.987116555s old)
2017/07/13 11:07:40 DEBUG : TV/: Attr:
2017/07/13 11:07:40 DEBUG : TV/: >Attr: attr=valid=1m0s ino=0 size=0 mode=drwxrwxr-x, err=
2017/07/13 11:07:40 DEBUG : TV/: ReadDirAll:
2017/07/13 11:07:40 DEBUG : TV/: >ReadDirAll: item=1772, err=
2017/07/13 11:07:40 DEBUG : TV/: Lookup: name=“Aaaaaaaah! (2015)”
2017/07/13 11:07:40 DEBUG : TV/: >Lookup: node=, err=no such file or directory
2017/07/13 11:07:40 DEBUG : TV/: Lookup: name=“Aaaaaaaah! (2015)”
2017/07/13 11:07:40 DEBUG : TV/: >Lookup: node=, err=no such file or directory
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/: >ReadDirAll: item=6, err=
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/: Attr:
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/: >Attr: attr=valid=1m0s ino=0 size=0 mode=drwxrwxr-x, err=
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/: ReadDirAll:
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/: >ReadDirAll: item=6, err=
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/: Lookup: name=“Abandoned Engineering - S01E01 - Silent Cities [HDTV-1080p x264 AAC].mp4”
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/: >Lookup: node=TV/Abandoned Engineering/Abandoned Engineering - S01E01 - Silent Cities [HDTV-1080p x264 AAC].mp4, err=
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/Abandoned Engineering - S01E01 - Silent Cities [HDTV-1080p x264 AAC].mp4: Attr:
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/Abandoned Engineering - S01E01 - Silent Cities [HDTV-1080p x264 AAC].mp4: >Attr: a=valid=1m0s ino=0 size=1082834784 mode=-rw-rw-r–, err=
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/: Attr:
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/: >Attr: attr=valid=1m0s ino=0 size=0 mode=drwxrwxr-x, err=
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/: ReadDirAll:
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/: >ReadDirAll: item=6, err=
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/: Lookup: name=“Abandoned Engineering - S01E02 - Mission- Space [HDTV-1080p x264 AAC].mp4”
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/: >Lookup: node=TV/Abandoned Engineering/Abandoned Engineering - S01E02 - Mission- Space [HDTV-1080p x264 AAC].mp4, err=
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/Abandoned Engineering - S01E02 - Mission- Space [HDTV-1080p x264 AAC].mp4: Attr:
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/Abandoned Engineering - S01E02 - Mission- Space [HDTV-1080p x264 AAC].mp4: >Attr: a=valid=1m0s ino=0 size=1071831370 mode=-rw-rw-r–, err=
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/: Lookup: name=“Abandoned Engineering - S01E03 - Lost Bridges [HDTV-1080p x264 AAC].mp4”
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/: >Lookup: node=TV/Abandoned Engineering/Abandoned Engineering - S01E03 - Lost Bridges [HDTV-1080p x264 AAC].mp4, err=
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/Abandoned Engineering - S01E03 - Lost Bridges [HDTV-1080p x264 AAC].mp4: Attr:
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/Abandoned Engineering - S01E03 - Lost Bridges [HDTV-1080p x264 AAC].mp4: >Attr: a=valid=1m0s ino=0 size=1087728668 mode=-rw-rw-r–, err=
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/: Lookup: name=“Abandoned Engineering - S01E04 - Ghost Ships [HDTV-1080p x264 AAC].mp4”
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/: >Lookup: node=TV/Abandoned Engineering/Abandoned Engineering - S01E04 - Ghost Ships [HDTV-1080p x264 AAC].mp4, err=
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/Abandoned Engineering - S01E04 - Ghost Ships [HDTV-1080p x264 AAC].mp4: Attr:
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/Abandoned Engineering - S01E04 - Ghost Ships [HDTV-1080p x264 AAC].mp4: >Attr: a=valid=1m0s ino=0 size=1078035834 mode=-rw-rw-r–, err=
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/: Lookup: name=“Abandoned Engineering - S01E05 - Quest for Power [HDTV-1080p x264 AAC].mp4”
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/: >Lookup: node=TV/Abandoned Engineering/Abandoned Engineering - S01E05 - Quest for Power [HDTV-1080p x264 AAC].mp4, err=
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/Abandoned Engineering - S01E05 - Quest for Power [HDTV-1080p x264 AAC].mp4: Attr:
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/Abandoned Engineering - S01E05 - Quest for Power [HDTV-1080p x264 AAC].mp4: >Attr: a=valid=1m0s ino=0 size=1084159341 mode=-rw-rw-r–, err=
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/: Lookup: name=“Abandoned Engineering - S01E06 - Roads to Nowhere [HDTV-1080p x264 AAC].mp4”
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/: >Lookup: node=TV/Abandoned Engineering/Abandoned Engineering - S01E06 - Roads to Nowhere [HDTV-1080p x264 AAC].mp4, err=
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/Abandoned Engineering - S01E06 - Roads to Nowhere [HDTV-1080p x264 AAC].mp4: Attr:
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/Abandoned Engineering - S01E06 - Roads to Nowhere [HDTV-1080p x264 AAC].mp4: >Attr: a=valid=1m0s ino=0 size=1080404067 mode=-rw-rw-r–, err=
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/: Attr:
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/: >Attr: attr=valid=1m0s ino=0 size=0 mode=drwxrwxr-x, err=
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/: ReadDirAll:
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/: >ReadDirAll: item=6, err=
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/: Attr:
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/: >Attr: attr=valid=1m0s ino=0 size=0 mode=drwxrwxr-x, err=
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/: ReadDirAll:
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/: >ReadDirAll: item=6, err=
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/: Attr:
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/: >Attr: attr=valid=1m0s ino=0 size=0 mode=drwxrwxr-x, err=
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/: ReadDirAll:
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/: >ReadDirAll: item=6, err=
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/: Attr:
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/: >Attr: attr=valid=1m0s ino=0 size=0 mode=drwxrwxr-x, err=
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/: ReadDirAll:
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/: >ReadDirAll: item=6, err=
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/: Attr:
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/: >Attr: attr=valid=1m0s ino=0 size=0 mode=drwxrwxr-x, err=
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/: ReadDirAll:
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/: >ReadDirAll: item=6, err=
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/: Attr:
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/: >Attr: attr=valid=1m0s ino=0 size=0 mode=drwxrwxr-x, err=
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/: ReadDirAll:
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/: >ReadDirAll: item=6, err=
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/: Attr:
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/: >Attr: attr=valid=1m0s ino=0 size=0 mode=drwxrwxr-x, err=
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/: ReadDirAll:
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/: >ReadDirAll: item=6, err=
2017/07/13 11:07:41 DEBUG : TV/1812 Napoleonic Wars in Russia/1812- Napoleonic Wars in Russia - S01E03 - Borodino [WEBDL-1080p x264 AAC [EN] eztv].mkv ®: Read: len=131072, offset=606535680
2017/07/13 11:07:41 DEBUG : TV/1812 Napoleonic Wars in Russia/1812- Napoleonic Wars in Russia - S01E03 - Borodino [WEBDL-1080p x264 AAC [EN] eztv].mkv ®: >Read: read=131072, err=
2017/07/13 11:07:41 DEBUG : TV/: Attr:
2017/07/13 11:07:41 DEBUG : TV/: >Attr: attr=valid=1m0s ino=0 size=0 mode=drwxrwxr-x, err=
2017/07/13 11:07:41 DEBUG : TV/: ReadDirAll:
2017/07/13 11:07:41 DEBUG : TV/: >ReadDirAll: item=1772, err=
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/: Attr:
2017/07/13 11:07:41 DEBUG : TV/Abandoned Engineering/: >Attr: attr=valid=1m0s ino=0 size=0 mode=drwxrwxr-x, err=
2017/07/13 11:07:41 DEBUG : TV/: Lookup: name=“Aashiqui.in (0)”
2017/07/13 11:07:41 DEBUG : TV/: >Lookup: node=, err=no such file or directory
2017/07/13 11:07:41 DEBUG : TV/: Lookup: name=“Aashiqui.in (0)”
2017/07/13 11:07:41 DEBUG : TV/: >Lookup: node=, err=no such file or directory
2017/07/13 11:07:41 DEBUG : TV/Adventure Time/Adventure Time - S08E09 - Islands, Part 3- Mysterious Island [WEBDL-1080p x265 AAC [EN] UTR].mkv ®: >Read: read=66781, err=
2017/07/13 11:07:41 DEBUG : TV/Adventure Time/Adventure Time - S08E09 - Islands, Part 3- Mysterious Island [WEBDL-1080p x265 AAC [EN] UTR].mkv ®: Flush:
2017/07/13 11:07:41 DEBUG : TV/Adventure Time/Adventure Time - S08E09 - Islands, Part 3- Mysterious Island [WEBDL-1080p x265 AAC [EN] UTR].mkv ®: >Flush: err=
2017/07/13 11:07:41 DEBUG : TV/Adventure Time/Adventure Time - S08E09 - Islands, Part 3- Mysterious Island [WEBDL-1080p x265 AAC [EN] UTR].mkv ®: Release:
2017/07/13 11:07:41 DEBUG : TV/Adventure Time/Adventure Time - S08E09 - Islands, Part 3- Mysterious Island [WEBDL-1080p x265 AAC [EN] UTR].mkv: ReadFileHandle.Release closing
2017/07/13 11:07:41 DEBUG : TV/Adventure Time/Adventure Time - S08E09 - Islands, Part 3- Mysterious Island [WEBDL-1080p x265 AAC [EN] UTR].mkv ®: >Release: err=
2017/07/13 11:07:41 DEBUG : TV/Adventure Time/: Lookup: name=“Adventure Time - S08E10 - Islands, Part 4- Imaginary Resources [WEBDL-1080p x265 AAC [EN] UTR].mkv”
2017/07/13 11:07:41 DEBUG : TV/Adventure Time/: >Lookup: node=TV/Adventure Time/Adventure Time - S08E10 - Islands, Part 4- Imaginary Resources [WEBDL-1080p x265 AAC [EN] UTR].mkv, err=
2017/07/13 11:07:41 DEBUG : TV/Adventure Time/Adventure Time - S08E10 - Islands, Part 4- Imaginary Resources [WEBDL-1080p x265 AAC [EN] UTR].mkv: Attr:
2017/07/13 11:07:41 DEBUG : TV/Adventure Time/Adventure Time - S08E10 - Islands, Part 4- Imaginary Resources [WEBDL-1080p x265 AAC [EN] UTR].mkv: >Attr: a=valid=1m0s ino=0 size=95293786 mode=-rw-rw-r–, err=
2017/07/13 11:07:41 DEBUG : TV/Adventure Time/Adventure Time - S08E10 - Islands, Part 4- Imaginary Resources [WEBDL-1080p x265 AAC [EN] UTR].mkv: Open: flags=OpenReadOnly
2017/07/13 11:07:41 DEBUG : TV/Adventure Time/Adventure Time - S08E10 - Islands, Part 4- Imaginary Resources [WEBDL-1080p x265 AAC [EN] UTR].mkv: >Open: fh=TV/Adventure Time/Adventure Time - S08E10 - Islands, Part 4- Imaginary Resources [WEBDL-1080p x265 AAC [EN] UTR].mkv ®, err=
2017/07/13 11:07:41 DEBUG : TV/Adventure Time/Adventure Time - S08E10 - Islands, Part 4- Imaginary Resources [WEBDL-1080p x265 AAC [EN] UTR].mkv ®: Read: len=131072, offset=0
2017/07/13 11:07:41 DEBUG : TV/: Attr:
2017/07/13 11:07:41 DEBUG : TV/: >Attr: attr=valid=1m0s ino=0 size=0 mode=drwxrwxr-x, err=
2017/07/13 11:07:41 DEBUG : TV/: ReadDirAll:
2017/07/13 11:07:41 DEBUG : TV/: >ReadDirAll: item=1772, err=
2017/07/13 11:07:41 DEBUG : TV/: Lookup: name=“Abandon (2002)”
2017/07/13 11:07:41 DEBUG : TV/: >Lookup: node=, err=no such file or directory
2017/07/13 11:07:41 DEBUG : TV/: Lookup: name=“Abandon (2002)”
2017/07/13 11:07:41 DEBUG : TV/: >Lookup: node=, err=no such file or directory
2017/07/13 11:07:42 DEBUG : TV/: Attr:
2017/07/13 11:07:42 DEBUG : TV/: >Attr: attr=valid=1m0s ino=0 size=0 mode=drwxrwxr-x, err=
2017/07/13 11:07:42 DEBUG : TV/: ReadDirAll:
2017/07/13 11:07:42 DEBUG : TV/: >ReadDirAll: item=1772, err=
2017/07/13 11:07:42 DEBUG : TV/: Lookup: name=“The Abandoned (2016)”
2017/07/13 11:07:42 DEBUG : TV/: >Lookup: node=, err=no such file or directory
2017/07/13 11:07:42 DEBUG : TV/: Lookup: name=“The Abandoned (2016)”
2017/07/13 11:07:42 DEBUG : TV/: >Lookup: node=, err=no such file or directory
2017/07/13 11:07:42 DEBUG : TV/: Attr:
2017/07/13 11:07:42 DEBUG : TV/: >Attr: attr=valid=1m0s ino=0 size=0 mode=drwxrwxr-x, err=
2017/07/13 11:07:42 DEBUG : TV/: ReadDirAll:
2017/07/13 11:07:42 DEBUG : TV/: >ReadDirAll: item=1772, err=
2017/07/13 11:07:42 DEBUG : TV/: Lookup: name=“ABBA”
2017/07/13 11:07:42 DEBUG : TV/: >Lookup: node=, err=no such file or directory
2017/07/13 11:07:42 DEBUG : TV/: Lookup: name=“ABBA”
2017/07/13 11:07:42 DEBUG : TV/: >Lookup: node=, err=no such file or directory
2017/07/13 11:07:42 DEBUG : TV/: Lookup: name=“ABBA”
2017/07/13 11:07:42 DEBUG : TV/: >Lookup: node=, err=no such file or directory
2017/07/13 11:07:42 DEBUG : TV/: Lookup: name=“ABBA”
2017/07/13 11:07:42 DEBUG : TV/: >Lookup: node=, err=no such file or directory
2017/07/13 11:07:42 DEBUG : TV/: Attr:
2017/07/13 11:07:42 DEBUG : TV/: >Attr: attr=valid=1m0s ino=0 size=0 mode=drwxrwxr-x, err=
2017/07/13 11:07:42 DEBUG : TV/: ReadDirAll:
2017/07/13 11:07:42 DEBUG : TV/: >ReadDirAll: item=1772, err=
2017/07/13 11:07:42 DEBUG : TV/: Lookup: name=“Abattoir (2016)”
2017/07/13 11:07:42 DEBUG : TV/: >Lookup: node=, err=no such file or directory
2017/07/13 11:07:42 DEBUG : TV/: Lookup: name=“Abattoir (2016)”
2017/07/13 11:07:42 DEBUG : TV/: >Lookup: node=, err=no such file or directory

calisro · July 13, 2017, 10:37am

Thanks. Besides the actual 403 message, what problem does this cause? My understanding of the Plex problem is a download limit and not an API limit.

aj1252 · July 13, 2017, 11:18am

It results in “403 Forbidden” and a 24-hour ban.

I have been trying to figure out what is causing it but it is not that easy since you have one try every 24 hours plus you usually do not usually want to ban yourself However, I think they operate with a limit shorter than the normal 100 seconds API one. Normally you have 1000 requests each 100 seconds on the API, which means you in principal can do 999 in 1 second and stay quiet the next 99 seconds without triggering the API limit. I think these API bursts are what’s creating the bans on drive.

Usually with the normal API limits you get warnings like “403: User Rate Limit Exceeded”, but with drive everything looks normal and poof your banned.

calisro · July 13, 2017, 11:44am

I see. My assumption was you were referring to 403 user rate limit exceeded. Thanks for the info.

aj1252 · July 13, 2017, 1:10pm

So with Plex, Sonarr and Radarr running simultaneous restricted to 1req/1sec I was not able to get a ban, regardless of how much I scaned and tried to push it to the limit.

Up next; lower to 1req/100ms which is the normal API limit but still 10 times less than the increased one I’m using.

ncw · July 13, 2017, 1:20pm

That is good news.

I coudl potentially allow a burst of up to N transactions if you haven’t used any for a while which might make it more usable.

What is the performance of actually playing video like?

Rclone’s google drive backend limits transactions to one every 10ms normally.

aj1252 · July 13, 2017, 1:30pm

At 1 second its will play ok, but it takes forever to start. Expecially if you have scanning and stuff running at the same time.
At 100ms the loading is acceptable.

Do you have more information on why 403 Forbidden is triggered? At 10ms even my 100req/sec API should never hit.

djsecrist · July 13, 2017, 4:13pm

This is very interesting and exciting! I’m going to risk a ban and try it out, too (with a conservative value). I think caching (like PlexDrive and node-gdrive-fuse) is the best ultimate solution, but this is a great idea that would probably work in conjunction with caching (to prevent bans while building the cache). My reasoning is that I would probably end up using this on multiple servers and wouldn’t want to cause a ban if two or more servers hit Google with high requests at the same time. So, I would probably need to keep this set to a conservative value, but I would still want maximum performance if only one server was actively using the cloud drive.

Thanks! Keep up the great work!

aj1252 · July 13, 2017, 4:32pm

I’ve been running everything against it for hours now, no ban.
So yeah its working great.
Currently at 100ms restriction using a 100req/sec API.

djsecrist · July 13, 2017, 6:35pm

I’ve been running it for a couple of hours now, too. I only have the default 1000req/100sec (i.e., 10req/sec) limit and am running it with a setting of --min-transaction-interval 200ms. If my math is correct, that should keep me below half of my limit since I will likely use it on my other server and need some margin for both. It seems to be working because the highest req/sec that I see in the Google Console is 3req/sec over the last hour. I have done a full Plex library refresh while running two streams concurrently. The scan definitely went a lot slower, but I don’t really care. The user experience in terms of starting, resuming, and seeking is not bad at all! Movies start up in about 4-6 seconds, which is about on par with PlexDrive and just a second or two slower than node-gdrive-fuse. I am really excited about this option, even without the caching! (For the record though, I still want caching…)

aj1252 · July 13, 2017, 10:53pm

Still running strong here at 100ms, no bans.
Reading and playing is fine.
rclone copy and move is very slow even at 1ns or the parameter completely omited.
Have not tested writing to the mount yet

ncw · July 13, 2017, 11:56pm

Excellent

Err, yes I made a mistake with the code so if you omit the parameter it still applies the filter… Not sure why it is slow with 1ns, but 0 should turn it off.

aj1252 · July 14, 2017, 12:03am

“rclone copy and move is very slow”
Let me test this some more tomorrow, thinking about it now, this might have been an issue at my end. I was throwing everything at the mount to see if it would stop, so I might have maxed out my connection.