How to optimize plex scanning for newer stuff on your drive

Hey guys, I have a pretty common plex, gsuite and rclone crypt setup running, but my media scanning is usually manual. So I add some stuff to gsuite through a cron job (rclone copy/move) and later I just ask Plex to scan my media.

Two problems emerge however. This method makes Plex scan the whole library, which can take a while.
Most of the times, it goes through the folders very quickly, dozens in a few seconds, and quickly reaches the new stuff, processing, matching it and adding to the library properly.

However, sometimes it insists on processing stuff thatā€™s been there for ages for no apparent reason, so this can take a long time.

Which scripts and tools are you guys using to avoid these hiccups?

Hereā€™s the mount systemd:

[Unit]
Description=Rclone Mount Gsuite
After=network-online.target
Wants=network-online.target
 
[Service]
Type=notify
ExecStart=/usr/bin/rclone mount gcrypt: /mnt/gsuite \
   --allow-other \
   --buffer-size 32M \
   --dir-cache-time 128h \
   --drive-chunk-size 32M \
   --log-file /home/rock64/RcloneMountLOG.log \
   --vfs-read-chunk-size 64M \
   --vfs-read-chunk-size-limit off \
   --rc 
ExecStop=/bin/fusermount -uz /mnt/gsuite
Restart=on-abort
User=rock64
Group=rock64

[Install]
WantedBy=gmedia.service

My plex settings disable automatic scans, thumbnails and all the usual stuff.

Thanks a lot for you input.

Define ā€˜quite awhileā€™.

Are you using your own client ID/API key?

https://rclone.org/drive/#making-your-own-client-id

I use plex autoscan to automate my scanning and such and it works great. It also has an option to emtpy the trash which if you have removed media, it takes longer too.

If my cache is up to date a plex scan takes ~10 seconds. If my cache is completely stale, it takes maybe 5-7 minutes as I have about 21k TV shows and 2k movies and a 54TB GD currently.

1 Like

Iā€™ll begin to use my own client ID/API key, thanks for the instructions.

When Plex doesnā€™t try to reprocess old things, itā€™s quite fast actually, in a handful of minutes newer stuff would come up. Watching the alerts, I can see it goes through movie and series folder at a reasonable pace of maybe 10 or more a minute untill it reaches newer stuff. If it begins to process old stuff (and thatā€™s what Iā€™m trying to avoid by having an intelligent script to focus scans in the correct places), it can go for hours probably if I donā€™t cancel.

When you mention cache, youā€™re talking about a cached remote or --dir-cache-time? I only use an encrypted remote and mount it directly in my single board computer running ubuntu.

dir-cache-time as that ā€˜cachesā€™ in memory the directory and file structure.

If everything is analyzed in Plex, it basically just does a file check.

I use this script if you wanted to check your plex analyzed stats.

At the end, it should have 0.

24462 files in library
0 files missing analyzation info
0 media_parts marked as deleted
0 metadata_items marked as deleted
0 directories marked as deleted
24009 files missing deep analyzation info.

0 files missing analyzation info is the key line from that as all my media is analyzed. The only reason things take ā€˜longerā€™ is if media isnā€™t analyzed as that normally happens when it is first added unless someone stopped it. ā€˜deep analysisā€™ is only used for bandwidth limiting so it isnā€™t needed.

Iā€™m on Windows, and Iā€™ve simply learned to live with the fact that Plex takes ages to scan a large movie folder. All my stuff is on G Drive, but the scan took just as long when I had everything on local hard drives, if I remember correctly. At close to 9000 movies and 1600 shows, my daily scan takes about two hours, if not longer. I believe this is due to the way Windows Explorer handles folders with a lot of sub-folders. The TV section scans significantly faster, though. My rclone mount settings are almost entirely default:

ā€“allow-other --buffer-size 256M --dir-cache-time 72h --read-only

My conf is:

[Google_Drive_Crypt]
type = crypt
remote = Google_Drive:crypt
filename_encryption = standard
directory_name_encryption = true

This results in fantastic performance for playback. If anyone has any suggestions on how to speed things up in Windows, Iā€™m all ears.

A daily scan taking 2 hours really means something is broken imo.

If everything is analyzed a scan should take minutes.

If the directory/file cache is out of date, it may take a little longer meaning 10 minutes.

You would think so, yes. There seems to be some kind of threshold at which Windows starts slowing down, for whatever reason. Could be 5000 folders, or something like that, because my TV section has not yet started to show these symptoms at 1600. After adding new stuff, the cache rebuilds quickly, like a minute perhaps for the movie section. And I always rebuild (as in click on the Movie and TV folder to show their updated contents) before I run a Plex scan.

I know that you and many others do not use Windows, but if someone with a similar setup as mine could tell us their experiencesā€¦ Iā€™ve been using rclone for years now, and everything is super stable, including the mount, which is basically up and running 24/7. I only take it down for a new beta, which is almost daily, of course :wink: But again, I believe this to be an issue with Windows, not rclone, WinFsp, or Plex.

Without plex scanning, how long does it take to run a recursive directory command on your GD mount? Iā€™m not sure on Windows how to time it other than just eyeballing a second clock.

I just ran ā€œrclone lsd remote:path -Rā€, and itā€™s taking about the same time as Iā€™d expect from running a Plex scan. About 3-5 folders per second, with a small pause in between.

No, thatā€™s not what I wanted to see as that is a ā€˜coldā€™ scan.

If you run the command from a dos prompt, twice, it should be ā€˜slowā€™ one time and fast the second time.

dir /s on your mounted folder.

Got it. Ran dir /s on the mount root. Took about 16 minutes to go through 126014 files/59100 folders. And from the looks of it, even though this was a mount I had established yesterday, it treated it as a cold scan. Subsequent scans are blazing fast, like 10 seconds for the above. So, maybe I have been doing this wrong the whole time:

  1. I establish the read-only mount.
  2. I start Plex.
  3. I upload stuff via RcloneBrowser outside of the mount.
  4. I establish a second, writeable mount, and I move the uploaded stuff into the proper folders. Then I stop the second mount.
  5. I use Windows Explorer to browse the existing (or sometimes newly established) read-only mount, going into the root library folders that Plex will scan.
  6. I start the Plex scan, which is always much, much faster on the TV sections than on the Movie sections, but still takes about two hours to complete.

Do you suggest I do a ā€œdir /sā€ on the mount every time right before a Plex scan?

Here a few things that I take into account for my library.

  • You can start the mount with --rc and I use this command to ā€˜primeā€™ the cache.
rclone rc vfs/refresh recursive=true
  • I break up my Radarr with stuff I never plan to upgrade again and I dump those all in a single folder. Plex supports that with movies but not TV shows.
felix@gemini:/gmedia/Movies$ ls | wc -l
1942

Thatā€™s roughly 2k files in a single folder.

I donā€™t think you have a Windows thing per se as itā€™s just a lot of API calls to list out every directory and it gets worse the more movies that you have.

Iā€™ll give that a try. Is that essentially the same as running ā€œdir /sā€ on the mount itself?

Itā€™s a little faster, but doesnā€™t matter that much as you can run the dir /s or just leave plex scan a few times a day to refresh the cache. I just leave mine at every 6 hours.

Just initiated another scan after running ā€œdir /sā€ on the mount. Iā€™m afraid there is absolutely no difference in speed. Plex scans about 2-3 movies per second. I really donā€™t think this has anything to do with rclone. Itā€™s either Plex or Windows, or both. I recall users on the Plex forum having this same issue once they had a substantial amount of sub-folders within the Movie folder.

Again, if there is anyone on Windows with a Movies folder and thousands of sub-folders, please tell us about your Plex scan experience.

Why not split it up and have a few larger folders and break it up a bit if you think thatā€™s the issue? No reason not to try to work around it. I only have ~2k movies so doesnā€™t quite scale up to your number so I canā€™t really test much.

I thought about doing that, but my OCD wonā€™t let me :scream:

Iā€™m OK with how things are for now, since I really only need to do one scan a day. Been following your advice for a long time, and Iā€™m basically using an even simpler version of your settings. So, thanks for that and all your help here!

Organize it, put A-F, G-blah, etc. You can definitely make it nice and organized and keep your OCD in check.

I might have to eventually. What gets me is that Kodi scanned that amount of folders in mere seconds, back when I used that.