What are optimized settings for Plex integration on MacOS

[deathley]

What is the problem you are having with rclone?

Error 403 keeps happening after 3h of Plex library sync:

Hi all, I've been trying to integrate my GD storage with Plex after successfully uploading my entire 19TB library. I've tried a few times (3 to be exact) to add the directories to Plex and let Plex perform library scans. It usually works well at first, but then, after 3h, i get errors 403.
I'm using a mount command which uses ~20% of my RAM, which does not seem to cause problems, i have API tokens in my config and my quotas are large (see info below).
I don't think what I'm trying to do is rocket science, but i can't, for the life of me, figure out what I'm doing wrong.

• Are there settings on Plex that I should use to disable multiple metadata requests?
• Are there other options missing in my config/mount command?

Thanks in advance...

What is your rclone version (output from rclone version)

v1.53.3

Which OS you are using and how many bits (eg Windows 7, 64 bit)

MacOS Big Sur (11.1) on 3.2GHz i7 / 32GB Ram / 2018 MacMini

Which cloud storage system are you using? (eg Google Drive)

Google Drive, for which i have credentials

The command you were trying to run

This is my rclone mount command

rclone mount GD:Storage ~/Desktop/gdrivetest \
--allow-non-empty \
--log-file ~/Documents/_FILES_HANDLING/rclone.log \
--vfs-read-chunk-size 2G \
--vfs-read-chunk-size-limit off \
--drive-chunk-size 2G \
--buffer-size 12G \
--allow-other \
--rc \
--allow-non-empty \
--dir-cache-time 96h \
--vfs-cache-mode writes \
--cache-dir  ~/Movies/cache \
--rc-addr REDACTED_IP:5572 \
--rc-pass REDACTED \
--rc-user REDACTED \
-vv 

The rclone config contents with secrets removed.

[GD]
type = drive
token = {"access_token":"REDACTED","token_type":"Bearer","refresh_token":"1REDACTED","expiry":"2021-03-07T15:13:48.992784Z"}
client_id = REDACTED.apps.googleusercontent.com
client_secret = REDACTED

A log from the command with the -vv flag

Sample log with 3x 403 errors

2021/03/07 14:11:04 DEBUG : TV/2 Broke Girls/2.Broke.Girls.S02.Season.2.Complete.720p.HDTV.X264-MRSK[cttv]/2.Broke.Girls.S02E14.720p.HDTV.X264-MRSK.mkv: >Open: fd=TV/2 Broke Girls/2.Broke.Girls.S02.Season.2.Complete.720p.HDTV.X264-MRSK[cttv]/2.Broke.Girls.S02E14.720p.HDTV.X264-MRSK.mkv (r), err=<nil>
2021/03/07 14:11:04 DEBUG : TV/2 Broke Girls/2.Broke.Girls.S02.Season.2.Complete.720p.HDTV.X264-MRSK[cttv]/2.Broke.Girls.S02E14.720p.HDTV.X264-MRSK.mkv: >Open: fh=&{TV/2 Broke Girls/2.Broke.Girls.S02.Season.2.Complete.720p.HDTV.X264-MRSK[cttv]/2.Broke.Girls.S02E14.720p.HDTV.X264-MRSK.mkv (r)}, err=<nil>
2021/03/07 14:11:04 DEBUG : &{TV/2 Broke Girls/2.Broke.Girls.S02.Season.2.Complete.720p.HDTV.X264-MRSK[cttv]/2.Broke.Girls.S02E14.720p.HDTV.X264-MRSK.mkv (r)}: Read: len=65536, offset=0
2021/03/07 14:11:04 DEBUG : TV/2 Broke Girls/2.Broke.Girls.S02.Season.2.Complete.720p.HDTV.X264-MRSK[cttv]/2.Broke.Girls.S02E14.720p.HDTV.X264-MRSK.mkv: ChunkedReader.openRange at 0 length 2147483648
2021/03/07 14:11:05 DEBUG : &{TV/2 Broke Girls/2.Broke.Girls.S02.Season.2.Complete.720p.HDTV.X264-MRSK[cttv]/2.Broke.Girls.S02E14.720p.HDTV.X264-MRSK.mkv (r)}: >Read: read=0, err=open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
2021/03/07 14:11:05 DEBUG : &{TV/2 Broke Girls/2.Broke.Girls.S02.Season.2.Complete.720p.HDTV.X264-MRSK[cttv]/2.Broke.Girls.S02E14.720p.HDTV.X264-MRSK.mkv (r)}: Flush: 
2021/03/07 14:11:05 DEBUG : &{TV/2 Broke Girls/2.Broke.Girls.S02.Season.2.Complete.720p.HDTV.X264-MRSK[cttv]/2.Broke.Girls.S02E14.720p.HDTV.X264-MRSK.mkv (r)}: >Flush: err=<nil>
2021/03/07 14:11:05 DEBUG : &{TV/2 Broke Girls/2.Broke.Girls.S02.Season.2.Complete.720p.HDTV.X264-MRSK[cttv]/2.Broke.Girls.S02E14.720p.HDTV.X264-MRSK.mkv (r)}: Release: 
2021/03/07 14:11:05 DEBUG : &{TV/2 Broke Girls/2.Broke.Girls.S02.Season.2.Complete.720p.HDTV.X264-MRSK[cttv]/2.Broke.Girls.S02E14.720p.HDTV.X264-MRSK.mkv (r)}: >Release: err=<nil>
2021/03/07 14:11:05 DEBUG : TV/2 Broke Girls/2.Broke.Girls.S02.Season.2.Complete.720p.HDTV.X264-MRSK[cttv]/2.Broke.Girls.S02E14.720p.HDTV.X264-MRSK.mkv: Attr: 
2021/03/07 14:11:05 DEBUG : TV/2 Broke Girls/2.Broke.Girls.S02.Season.2.Complete.720p.HDTV.X264-MRSK[cttv]/2.Broke.Girls.S02E14.720p.HDTV.X264-MRSK.mkv: >Attr: a=valid=1s ino=0 size=372686865 mode=-rw-r--r--, err=<nil>
2021/03/07 14:11:05 DEBUG : TV/2 Broke Girls/2.Broke.Girls.S02.Season.2.Complete.720p.HDTV.X264-MRSK[cttv]/2.Broke.Girls.S02E14.720p.HDTV.X264-MRSK.mkv: Open: flags=OpenReadOnly
2021/03/07 14:11:05 DEBUG : TV/2 Broke Girls/2.Broke.Girls.S02.Season.2.Complete.720p.HDTV.X264-MRSK[cttv]/2.Broke.Girls.S02E14.720p.HDTV.X264-MRSK.mkv: Open: flags=O_RDONLY
2021/03/07 14:11:05 DEBUG : TV/2 Broke Girls/2.Broke.Girls.S02.Season.2.Complete.720p.HDTV.X264-MRSK[cttv]/2.Broke.Girls.S02E14.720p.HDTV.X264-MRSK.mkv: >Open: fd=TV/2 Broke Girls/2.Broke.Girls.S02.Season.2.Complete.720p.HDTV.X264-MRSK[cttv]/2.Broke.Girls.S02E14.720p.HDTV.X264-MRSK.mkv (r), err=<nil>
2021/03/07 14:11:05 DEBUG : TV/2 Broke Girls/2.Broke.Girls.S02.Season.2.Complete.720p.HDTV.X264-MRSK[cttv]/2.Broke.Girls.S02E14.720p.HDTV.X264-MRSK.mkv: >Open: fh=&{TV/2 Broke Girls/2.Broke.Girls.S02.Season.2.Complete.720p.HDTV.X264-MRSK[cttv]/2.Broke.Girls.S02E14.720p.HDTV.X264-MRSK.mkv (r)}, err=<nil>
2021/03/07 14:11:05 DEBUG : &{TV/2 Broke Girls/2.Broke.Girls.S02.Season.2.Complete.720p.HDTV.X264-MRSK[cttv]/2.Broke.Girls.S02E14.720p.HDTV.X264-MRSK.mkv (r)}: Read: len=32768, offset=0
2021/03/07 14:11:05 DEBUG : TV/2 Broke Girls/2.Broke.Girls.S02.Season.2.Complete.720p.HDTV.X264-MRSK[cttv]/2.Broke.Girls.S02E14.720p.HDTV.X264-MRSK.mkv: ChunkedReader.openRange at 0 length 2147483648
2021/03/07 14:11:05 DEBUG : &{TV/3rd Rock from the Sun/Season 2/3rd Rock from the Sun - S02E10 - Gobble, Gobble, Dick, Dick SDTV.mp4 (r)}: >Read: read=0, err=open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
2021/03/07 14:11:05 DEBUG : TV/3rd Rock from the Sun/Season 2/3rd Rock from the Sun - S02E10 - Gobble, Gobble, Dick, Dick SDTV.mp4: Attr: 
2021/03/07 14:11:05 DEBUG : TV/3rd Rock from the Sun/Season 2/3rd Rock from the Sun - S02E10 - Gobble, Gobble, Dick, Dick SDTV.mp4: >Attr: a=valid=1s ino=0 size=91719510 mode=-rw-r--r--, err=<nil>
2021/03/07 14:11:05 DEBUG : &{TV/3rd Rock from the Sun/Season 2/3rd Rock from the Sun - S02E10 - Gobble, Gobble, Dick, Dick SDTV.mp4 (r)}: Read: len=65536, offset=0
2021/03/07 14:11:05 DEBUG : TV/3rd Rock from the Sun/Season 2/3rd Rock from the Sun - S02E10 - Gobble, Gobble, Dick, Dick SDTV.mp4: ChunkedReader.openRange at 0 length 2147483648
2021/03/07 14:11:05 DEBUG : &{TV/2 Broke Girls/2.Broke.Girls.S02.Season.2.Complete.720p.HDTV.X264-MRSK[cttv]/2.Broke.Girls.S02E14.720p.HDTV.X264-MRSK.mkv (r)}: >Read: read=0, err=open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded

Google API Quotas

Quota name	Limit
Queries per day	1,000,000,000
Queries per 100 seconds per user	10,000
Queries per 100 seconds	10,000

API Stats

RAM Usage

[St0rm]

In Plex > Settings ⇾ Scheduled Tasks > Perform extensive media analysis during maintenance > Uncheck

[deathley]

I have that unchecked already.
But i'm not sure it matters since i see those errors occur as early as the first Library Scan when I add mount folders to Plex.

[asdffdsa (jojothehumanmonkey)]

hi,

• update to latest stable, v1.54.0

• need to tweak plex settings, this was written by a fellow rcloner.
  https://github.com/animosity22/homescripts#plex-tweaks

• --drive-chunk-size
  as i understand that flag, each time rclone wants to read a single byte, it will download 2GB.
  for streaming, start with the default value

• --allow-non-empty
  unless you are 1000% sure, you know what this is, then remove it.
  also, that flag is used twice in the same command.

• 99.99% of the time, this is not a good idea
  https://rclone.org/drive/#drive-chunk-size

[deathley]

Thanks for your feedback.

@asdffdsa

• update to latest stable, v1.54.0

Will do.

@asdffdsa

• need to tweak plex settings, this was written by a fellow rcloner.
  https://github.com/animosity22/homescripts#plex-tweaks

I think the first tip was related to Linux only so doesn't really apply to MacOS?
I definitely have the 2nd setting properly set. But not sure i forgot about the first one when i added the library.

@asdffdsa

• --drive-chunk-size
  as i understand that flag, each time rclone wants to read a single byte, it will download 2GB.
  for streaming, start with the default value

I've seen on the forum that it downloads chunks of 2GB (in my particular use case) for files, until it reaches the total? i might be wrong
If i reduce to the default value (which is essentially 16 times less), it greatly increases buffering. I think my RAM can handle this. Also, the first time i tried to add library to Plex, i used the default value.

@asdffdsa

• --allow-non-empty
  unless you are 1000% sure, you know what this is, then remove it.
  also, that flag is used twice in the same command.

I've removed it now. Not sure what was going on but i would sometime get an error without this argument, telling me the folder wasn't empty, even though it clearly was. This doesn't seem to occur anymore.

@asdffdsa

• 99.99% of the time, this is not a good idea
  https://rclone.org/drive/#drive-chunk-size

Can you please clarify what exactly is not a good idea?

So, i tried this again, with an updated command removing only the allow-non-empty tag, with latest version, and making sure the settings were correct. It's been scanning my TV library for about 15 minutes without issues (longer than the last time i tried so good sign!)

[asdffdsa (jojothehumanmonkey)]

@asdffdsa

99.99% of the time, this is not a good idea

i meant that the size of the chunk is too large and also, i do not think it is needed.
i always use as few flags as possible, then if needed, tweak and add more flags.

this is by a fellow rcloner, with optimized settings for rclone mount
i suggest start with that, test and tweak and then tweak.

about plex scans, here is what i do, tho i use jellyfin.
unless you make the tweaks from that link i shared, you are going to have problems

1. prime the mount - rclone will scan the mount and cache folder/file structure and size and mod-time for files.
2. have jellyfin scan the mount of new/updated media. the scan will complete quickly.

to enable this, add --rc to the mount command.
to prime the mount, run this rclone rc vfs/refresh recursive=true
then have plex scan

[Animosity022]

@deathley

I've seen on the forum that it downloads chunks of 2GB (in my particular use case) for files, until it reaches the total? i might be wrong

Google drive (rclone.org)

That's only for uploading and has nothing to do with downloading.

[deathley]

Noted, thanks both.

I've removed that particular flag.
I also ran the prime command and am trying again (my last attempt failed at some point when i triggered a metadata refresh).

One question though:
what actually triggers the many scans to the mount from Plex? is it when metadata is fetched or only for thumbnail creation?

Edit: also, i've just realized my errors 403 coincide with a spike of 'drive.files.list' methods. is this normal?
I'm pretty sure that spike triggered when i clicked refresh all metadata on Plex as i was surprised that none of the shows had posters or were even matched

[VBB]

I would also turn off intro detection in Plex.

[deathley]

I still can't get this to work.
I have pretty much unchecked every box in the library settings.
I've updated my mount command to match recommended flags.
But i still get the 403 error.

I thought this was something standard that people were using rclone for but it seems overly complicated. Is there something I'm missing here?

--log-file ~/Documents/_FILES_HANDLING/rclone.log \
--vfs-read-chunk-size 2G \
--vfs-read-chunk-size-limit off \
--buffer-size 12G \
--allow-other \
--rc \
--dir-cache-time 96h \
--vfs-cache-mode writes \
--cache-dir  ~/Movies/cache \
--rc-addr REDACTED:5572 \
--rc-pass REDACTED \
--rc-user REDACTED \
-vv

[asdffdsa (jojothehumanmonkey)]

• 403 errors can take 24 hours to reset.

this is the goto guide for plex and rclone settings.
https://github.com/animosity22/homescripts/blob/master/systemd/rclone.service

• use --vfs-cache-mode full, it make use of sparse files
• remove:
  --vfs-read-chunk-size 2G
--vfs-read-chunk-size-limit off

[deathley]

Thanks, will try that.

I'll also add --poll-interval 15s , seems to provide improvement to API quota hits.

[deathley]

Hi all, i've tried again, and am not seeing the error 403 occur anymore.
The first time i tried, Plex just hung, and local libraries got unavailable until i quit the app and re-opened (which obviously stopped the scans).
Upon retrying, it worked pretty well.
When i triggered a metadata refresh, i did not get more errors, which is a good sign.
I'll keep monitoring for errors and will request to close this once my library is fully added without errors.

[deathley]

I'm not seeing any more errors but my mount now exits after a while (happened twice, second time after 2h40m)?

2021/03/09 01:42:15 DEBUG : /TV/mixed-ish/Season 1/mixed-ish - S01E11 - When Doves Cry WEBDL-1080p.mkv: >Read: n=4096
2021/03/09 01:42:15 DEBUG : Google drive root 'Storage': Destroy: 
2021/03/09 01:42:15 DEBUG : Google drive root 'Storage': >Destroy: 
2021/03/09 01:42:15 DEBUG : Not calling host.Unmount as mount already Destroyed
2021/03/09 01:42:15 DEBUG : vfs cache: cleaner exiting
2021/03/09 01:42:15 DEBUG : Unmounted successfully

[ncw (Nick Craig-Wood)]

@deathley

I'm not seeing any more errors but my mount now exits after a while (happened twice, second time after 2h40m)?

This is a macOS thing - the mount exiting for no reason. I've attempted to fix this in v1.54.1

[deathley]

Thanks, i've downloaded 1.54.1 and will try this again.
Happy to be your tester

[ncw (Nick Craig-Wood)]

@deathley

Happy to be your tester

I'm hoping that the patch in 1.54.1 (which is very simple, just changing a default from 15m to 10m) will fix the problem, but I'm not sure! It did work for someone else so it might work for you.

[deathley]

Yup looks like it’s working alright!

However, my library sync is not behaving very well. I don’t any errors per se but the stuff I saw added in the library yesterday before going to bed are not showing up anymore. And when I scan the library again, it works away (I can see stuff being pinged in the rclone log) but the entire PMS becomes really slow.
I’m not sure if there’s any relevance in sharing logs, but are there tips on standard use cases you guys might have?

[deathley]

Hi all,

I'm probably going to give up on this. I stil have no idea what I'm doing wrong, i can see I'm close but there's a lot of hurdles to get this to work. Every time i try to add my library, when i don't get errors 403, the entire PMS starts hanging after a few hours (even though i have everything pretty much turned off in the library settings), and i need to restart it, and re-scan, etc...
I also tried to use Google Drive Desktop and the same thing is happening, which indicates it's probably not related to rclone, so maybe my host or bandwidth...

Unless anyone has any last minute advice, I'm probably going to let this close automatically and will revert to local storage, even though i have unlimited GDrive storage.
It's a shame but i just can't make it work no matter what i try...

[asdffdsa (jojothehumanmonkey)]

what is the rclone mount command?

this might be a work around
rclone serve dlna