Rclone cache, plex, emby, encfs, unionfs with 100TB+ on gdrive vs plexdrive

everything is working fine with plexdrive in this setup aside from emby scan times… but i’m not sure if that would be improved at all with rclone/rclone cache…

anyone have any experience with rclone cache and emby, particularly scan times?

What is your scan time and what version of Emby Server?

its the INITIAL scans that i wonder about…

once the bulk is re-imported (i’m using the latest beta which is lightning fast really) a full scan is now about as quick as plexs (45 min to an hour)

its the initial scans when i do a rebuild (like i am having to to take advantage of the new databases speed improvements) that are the issue… on plex the initial re-import takes a full day say for the tv library… where with emby its like 5-10 times that :frowning:

100TB of media will definitely take more than a day.

I’ve scanned ~40TB in ~2-3 days.

Basically, cache would be slower for scanning as the vfs-chunk-read-size would be faster.

Both Plex and Emby do a ffprobe type command on each file and you can test/simulate something like that by going a mediainfo on a file. Depending on the file and how many subs/audio/etc it has, it can take anywhere from 1-2 seconds to 10-15 seconds per file.

Multiply that out for every file for the initial seed of your library and you’d have how long it takes.

Once it’s scanned, my Emby takes a 2-3 minutes to do a full scan and Plex is much faster as it takes 20-30 seconds at most.

1 Like

Do you let Emby save the metdata in NFO files? This should speed up things quite a lot.

@Animosity022, for me the daily full scan takes under 1 minute, usually 30-45 seconds. I do not know if it was the vfs-cache that picked up the pace or newer server versions or maybe even a combination of both but it definitely got way faster over the last months.

no, because it can’t write to the unionfs mount.

geez… minute scans.,… hahaha…

before i decided to try the new betas/new db format, when i had all 100tb+ imported and was only adding a hundred things a day maybe… the fastest library scan i ever had with emby was 45 minutes… and i’ll take that in a heartbeat : ) it used to take over 24 hours even after my initial scan : ) :frowning:

i just wish i knew where exactly the slowdown is on the initial importing/scan… sometimes it seems like emby has to download the ENTIRE file to analyze something… i’ts done over like 30tb of transfer just for the re-importing and i’m not done… : ( during the initial scan it averages 80mbs a sec downloading… but still seems to crawl at a glacial pace…

love the program to bits… but hate the endless initial scans. and unlike plex you really have no idea what its doing or if its stuck… days go by and yer not sure if its even doing anything hahaha…

That’s strange, what settings do you use?

My settings are the same, but plex would take a day or two to rescan everything while emby would be done in an hour… - different servers.

Well, if you add hundreds of stuff every day then it’s quite normal that it takes a bit to scan the files and if you do extract chapter images then it take even more.

I am not sure if you will be able to speed things up much more in your case.

i’m wondering if it has anything to do with the unionfs layer i have merging local and remote folders…

i don’t really need that… trying to figure out how to edit my commands to have the decrypted media path be the same as it is now in plex and emby and removing the merged uionfs layer…

I do not save NFOs but an extra minute or so doesn’t really matter to me:

1 Like

i tried removing the unionfs local/gdrive merge layer…
and mounting to my media folder… alas although my ubuntu user can see the files just fine… i cannot for the life of me get plex or emby to see them : (

tried doing the 2 or 3 plex and emby changes to run as my user… but could never get it to work…

does anyone know how to plexdrive mount and let plex and emby see the files?? i have the allow other option in there and it worked fine for the union merged folder… alas with that layer removed and just the media fodler plexdrive mounted… neither plex nor emby can see it… arghh… plexdrive is no longer supported anywhere… having a hard time figuring out where/who to ask : (

but why do you scan every day your plex library and not only the new content and for the newly added stuff the subtitle and thats it?! why do you stress your gdrive files all the time?

initial load OK, which I have maybe every 2 years
but not all the time either

well i add hundreds of things a day usually… i do just scan for new stuff… and the plex scans are superfast… just a few minutes… it’s the initial scans i’m driven crazy by… i’m on like week number 5 or 6 of a reimport/rebuild of my emby library hahaha… made necessary by having to nuke the database and start fresh to use the much faster new database system in the newer betas… : ) : (

the scans are just interminable… weeks… weeks…

i’m sure there is a way to speed it up… plexdrive or unionfs must be the culprit… but cant figure out how to successfully remove the unionfs layer and still have plex and emby see the media mount… or what if any commands i could add to my plexdrive command to make it better/faster…

one of the reasons i headed back to this forum for the first time in a while was to see the state of rclone cache and see if i could switch back… but mostly what i’m hearing is that with a library as big as mine it would be slower scans with cache… :frowning:

IIRC using nfo’s should speed up the library rebuild from scratch.

yes… it would… but all my files are on google drive and emby can’t write to the media folder : ) : (

Yes, I assure you Emby can write directly to a mount, it’s the way it’s working over here.
What does your mount command look like?

well i came back to these forums last week thinking of trying to switch back to rclone if the cache was working well enough now… but my main problem/reason were insanely long initial scan times in emby and several people here told me the rclone cache setup would actually be a little slower for some reason : (

i tried cutting out the unionfs merged folder layer to see if it was a problem and also to let emby write to the mount (it cant write to the merged fuse mount) by going from

plexdrive mount /home/xxx/plex2/.gdrive -o allow_other /home/xxx/plex2/gdrive
ENCFS6_CONFIG=’/home/xxx/plex2/.encfs6.xml’ encfs /home/xxx/plex2/.gdrive /home/xxx/plex2/gdrive
ENCFS6_CONFIG=’/home/xxx/plex2/.encfs6.xml’ encfs /home/xxx/plex2/.local /home/xxx/plex2/local
unionfs-fuse -o cow -o allow_other /home/xxx/plex2/local=RW:/home/xxx/plex2/gdrive=RW /home/xxx/plex2/media/

to-

plexdrive mount /home/xxx/plex2/.gdrive -o allow_other /home/xxx/plex2/media
ENCFS6_CONFIG=’/home/xxx/plex2/.encfs6.xml’ encfs /home/xxx/plex2/.gdrive /home/xxx/plex2/media
ENCFS6_CONFIG=’/home/xxx/plex2/.encfs6.xml’ encfs /home/xxx/plex2/.local /home/xxx/plex2/local

but try as i might although my regular ubuntu user could see the media mount just fine, emby and plex just could not see it (tried changing the plex or emby user following 3 or 4 steps for both but that was unsuccessful as well) : (

You are talking about rclone but those two mounts are using plexdrive…

Anyway, in the second one you are missing the allow other flag and that’s most probably why Emby is not seeing the mount as it simply doesn’t have the permission.

missing? it’s there in the plexdrive command

it’s just there is no unionfs merged media folder in the second one, just the mounted gdrive media folder with no merging/unionfs

something i must be missing to really allow other without the unionfs merged mount though… i just can’t figure it out…

Ohh sorry, I oversaw it.
Well, from my side I can’t help you further. Sorry!

1 Like