Rclone Cache + Plex - Long Scan & Long Load

Help and Support
1

Hey all!

I have about 30 TV Shows on my Google Drive currently that is uploaded through rclone crypt.

I'm currently scanning them from an rclone mount: drive -> cache -> crypt.

It took about 12 hours to scan everything which seems really slow. If I click play, it takes 30 seconds - 2 minutes to actually play. However, once it starts playing, it doesn't buffer and I can seek with little to no problems. This is all direct playing.

I have turned off video preview thumbnails in the library settings, and unchecked all the scan options in the settings.

OS is Windows 10 using the latest rclone beta. I understand that Linux is preferred but this server is also used for games and Linux isn't there yet.

Plex is installed on an NVMe SSD and the rclone mount and cache are pointed to a 6TB HDD. The HDD is dedicated for the cache and mount so I let it take up 2.5TB (not sure if this is correct).

The cache has:

chunk_size = 5M
info_age = 1d
chunk_total_size = 2.500T

The command mount is:

rclone mount --read-only --allow-other --allow-non-empty -vv --buffer-size=0M --vfs-read-chunk-size=128M --vfs-read-chunk-size-limit=2G --cache-workers=10 --cache-chunk-path E:/Cache/Chunk --cache-db-path E:/Cache/DB Drive_Crypt: E:/First

Checking task manager whenever it scans/starts-up, it uses around 5% CPU, 20% ram and 1% network so I don't see what the issue is. My internet is 110 down and 10 up.

As you can see, I've been playing around with the flags but alas, it still is giving me problems.

Any help would be appreciated.

2

You seem to be mixing the rclone cache and the vfs configs together.

If you want to just use VFS, you can use a config I have here:

Or simply just use:

rclone mount gcrypt: /GD \
   --allow-other \
   --read-only \
   --dir-cache-time 48h \
   --vfs-read-chunk-size 16M \
   --vfs-read-chunk-size-limit 2G \
   --buffer-size 512M \
   --log-level INFO

That says keep a cache of the directory and file names for 48 hours. Start off with a chunk size of 16M and allow it to grow when it wants to read up to 2G and keep a buffer-size of 512M in memory. This helps with direct playing and scanning.

So you want to configure in your rclone conf the root remote (GD) and a crypt on top of that and mount the crypt basically removing the cache from the equation. You can leave your .rclone.conf having both if you want to test later or change things around.

2 Likes
3

Thanks for the response.

I’ve tried your settings and it works much faster. It is still taking around 10-20 seconds to load things but I can probably play around with it to see.

Looking at the Plex logs, it does an initial scan then runs another three for the same item despite me turning off thumbnails.

I did get an out of memory error when I tried scanning a ~500 episode show. I have 16GB ram. Any ideas?

4

By all means, turn down this line as I just took a stab based on your config:

–buffer-size 512M to maybe --buffer-size 100M

Plex opens a lot of files at a time if you haven’t scanned the library before so you will consume that much memory per file opened so you can always bump it up once you stabilize. That’s not really an rclone, but a Plex thing.

1 Like
5

So I left it for a few hours at 100MB and it was still scanning. I come back and got the out of memory error. Checking the rclone logs with -vv shows that its still scanning episodes that should be finished. These are my Plex settings so I’m not sure if this is a Plex issue or an rclone one.

3

1
1.jpg1324×870 354 KB
2
2.PNG999×756 523 KB

6

It’s more a Plex question at this point.

Once you add a library and say it has 500 files. The first time you add it plex has to collect the video/audio information for the file and that it what plex calls analyzing a file and that’s after the file has been ‘scanned’ by the library.

If you turn on the debug logs, you can see quite the bit of detail in your plex logs directory.

In my case, I added a file and you can see an entry in my logs for the analyze of that file:

felix@gemini:/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Logs$ tail 'Plex Media Scanner Analysis.log'
Jul 18, 2018 18:18:02.307 [0x7f4baefff700] INFO - Plex Media Scanner v1.13.4.5251-2e6e8f841 - ubuntu PC x86_64 - build: linux-ubuntu-x86_64 - GMT -04:00
Jul 18, 2018 18:18:02.308 [0x7f4baefff700] INFO - Linux version: 4.15.0-24-generic (#26-Ubuntu SMP Wed Jun 13 08:44:47 UTC 2018), language: en-US
Jul 18, 2018 18:18:02.308 [0x7f4baefff700] INFO - Processor Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz
Jul 18, 2018 18:18:02.308 [0x7f4baefff700] INFO - /usr/lib/plexmediaserver/Plex Media Scanner --analyze --log-file-suffix  Analysis --item 24097

If you look at the scanner file, you can see the analyze happening:

felix@gemini:/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Logs$ tail -f  'Plex Media Scanner.log'
Jul 18, 2018 18:18:23.066 [0x7fd5c87ff700] INFO - Plex Media Scanner v1.13.4.5251-2e6e8f841 - ubuntu PC x86_64 - build: linux-ubuntu-x86_64 - GMT -04:00
Jul 18, 2018 18:18:23.066 [0x7fd5c87ff700] INFO - Linux version: 4.15.0-24-generic (#26-Ubuntu SMP Wed Jun 13 08:44:47 UTC 2018), language: en-US
Jul 18, 2018 18:18:23.066 [0x7fd5c87ff700] INFO - Processor Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz
Jul 18, 2018 18:18:23.066 [0x7fd5c87ff700] INFO - /usr/lib/plexmediaserver/Plex Media Scanner --analyze --item 24096

Analyze happens just one time, but it’s time consuming. I scanned a 45TB plex library in about 2 days. Plex opens up many files (20-35) at a time while this is going on and that’s what is probably eating up your memory.

You can lower the buffer size to 16M or something small while this is happening if you are memory constrained.

I usually turn of empty trash as well as you’d have to repeat this process if you empty the trash after a scan and the file was not present.

7

Well this library is currently sitting at 1.07TB and it still takes hours to scan. It scans pretty decently but it spends hours processing files for some reason.

Regarding the memory issue, I was watching my memory and it seems to crash at ~1.68GB despite me have 16GB. So it still seems like an rclone issue at least partially.

I’m starting to suspect that it is probably a Windows issue and I should just install Ubuntu on a VM and host Plex that way.

8

Sorry, I’m a Linux/Mac guy here. That does seem strange as having the buffer size low shouldn’t cause it to crash.

1 Like
9

You did help me with a few of the issues so I appreciate that. I’ll play around more with it. If it doesn’t work, I’ll just use a VM.

10

First thank you for this. I am working on a new rclone mount setup without plexdrive and am looking to optimize.

Right now my initial scan of my TV folder ~5TB is taking a long time (hours) which I believe is normal. Once that is complete I am going to compare vfs to a cache mount.

Looking at your github I have a few questions.

Are you still using the rclone-cache.service for anything or has it been depricated for gmedia-rclone.service?

What is the purpose of the gmedia-find.service?

Thanks!

11

My rclone-cache is just something I keep for testing cache issues more or less and I don’t actually use and you are correct, I use gmedia-rclone as part my overall gmedia.service.

gmedia-find just walks through all the directories and primes the cache. To be honest, it isn’t really needed as the first scan via plex or something else would do it anyway. For me, it also just spits out a number of files and if I really wanted to be sure, I could throw an alert if it was less than a certain number as well.

My stuff is pretty stable these days so I don’t have many issues in regards to that.

12

Ok thanks for the info! Once my initial TV scan is done I am going to swap over to your VFS setup and see how it goes.

Is it normal for the first scan to take a really long time? My assumption is its having to read a part of each file to check format, bitrate etc which would explain it. I am seeing a average 80Mb/s transfer rate on my server since I kicked off the import and have so far pulled down ~60GB.

13

I think I’ve done 45TB in like 1 1/2 days or so so 5TB shouldn’t be that bad. I would think 4-8 hours or so. The vfs-setup definitely moves faster with scanning than the rclone cache by anywhere from 5-10 seconds per file.