So im using rclone in vfs cache mode full so i can build a cache for my server and all the other things scanning it. From what i understand with the cache mode on full it should cache in chunks the most used parts of the files (like for plex and emby scans). It seems to be doing this but im having a couple performance issues that have me confused. I assumed that since that cache was on local disk and my dir cache time high that i should be able tog et basically local storage list times on the mounted directory. but when i do ls -R to the mounted directory it takes a quite a long time. However if i do the same thing to the directory where the cache is stored it takes almost no time at all.
Shouldnt these times actually be a lot closer?
rclone should be serving from the stored data in the cache and using the dir cache for this before hitting the remote right?
i would think that because these are stored locally and shouldnt need to be requested from the remote that the times should be just slightly slower then if listing the actual directory where the cache is.
Any and all help would be greatly appreciated.
Also using the vfs cache for very specific reasons and cant not use it. The data from scans and most frequently played parts of items really need to be cached locally.
Happy Holidays and thank you for your time!
What is your rclone version (output from rclone version)
os/version: ubuntu 18.04 (64 bit)
os/kernel: 4.15.0-163-generic (x86_64)
Which cloud storage system are you using? (eg Google Drive)
The command you were trying to run (eg rclone copy /tmp remote:tmp)
when trying to fun this command i get this output
2021/12/25 20:13:02 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "rc" "vfs/refresh" "recursive=true" "--fast-list" "--rc-addr=:5576" "-vv"]
2021/12/25 20:18:02 DEBUG : 4 go routines active
2021/12/25 20:18:02 Failed to rc: connection failed: Post "http://localhost:5576/vfs/refresh": net/http: timeout awaiting response headers
this command did work and i do indeed get better listing times, however this with plex this hasnt seemed to make a difference. The scan time when using vfs with no cache on the same library is much faster and completes in minutes rather than hours. This just doesnt seem right because i can see that chunks have been stored on my local disk but am still not getting great times on the scan. I would think this would be much better scanning off the cache with the data stored on local disk as opposed to having to download from the Gdrive first.
Am i doing something wrong with my set up?
Cache or no cache, plex has to analyze files. If you change paths/locations, it has to analyze them and update them.
Are you using an encryption or just mounting directly as you posted a snippet of your rclone.conf so I can't tell
How many items are you scanning? What's the size of the drive? In the Linux world with Plex, analyzed media scans are pretty much instant as it takes a few seconds to scan my library.
You have a pleothra of flags too.
This one allows for over mounting and can cause issues. Should only use this if you have a very specific use case requiring it.
Just remove them these and use the defaults.
Are you mounting it read only for a reason?
Any reason for these? It not, I'd remove and use the defaults.
This goes back to the read only question as this flag is only for writing files and if you are ready only, why are you setting this?
Are you using a service account? You can remove your client ID/client secret and token and that's not needed if you have a service account. Those values are needed if you are not using a service account. A service account uses it's own client ID/secret.
Use this command not
The aysnc true runs it in the background and fast-list is redundant as you already have recursive there. That way you don't worry about the time out and it'll just finish on its own.
The paths havent been changed, i have a plex server on a similar system set up with regular vfs no cache and another server that im using to test with the cache.
Cache has been populated by a few scans. im just mounting directly with no encryption.
The library has 43,000 movies. The drive is very large. doing an rclone size now. The stored size of chunks cant be too big or else i wont have enough storage. it needs to be at least big enough for all the scan data and some of the most played files. which it currently is but doubling chunk size could be problematic. Also a big data waste.
What are the defaults for buffer size and chunk size limit?
also the defaults for vfs read ahead?
the goal is to find a sweet spot here. There is going to be several constant streams from my friends and family who are all major binge watchers. between that, and all the scanning form plex, emby, jellyfin, sonarr, radarr, etc all hitting this it will be a bit busy all the time.
i sk for the defaults because i dont want the system to get stressed and be able to handle 15 streams with multiple things scanning without issues. as it does for the most part with regular vfs.
also shouldnt all these options come after the data stored locally in the cache once its there. The data in the cache should be accessed before rclone starts downloading chunks from the remote right?
i can remove allow non empty and the transfers flags because indeed they are not needed with the current set up.
i can get you a log but was having trouble getting it pasted into places if it was longer then what i sent.
ok so then if the cache is populated already with data from a few scans then i shouldnt need to worry about the chunk size and the other flags at the moment because they shouldnt be getting hit when plex scans again and there is no new content.
It should be just hitting the cache which should be much faster then a similar mount with no cache. But that isnt happening. the regular vfs no cache is out performing when downloading it from the remote directly vs when accessing the already populated cache.
one scans in a few minutes and the other in 2 to 3 hours.
the path isnt changing though...... nothing is being uploaded to this drive, everything is being uploaded to another drive. so i mount it to same spot everytime and nothing is being moved or uploaded to it.
i also already have all the analysis things turned off in plex.....
its already gone through the scan multiple times now and the paths arent changing but its still performing less well then vfs with no cache by a large margin.
please post both rclone mount commands, one with vfs file cache, one without the vfs file cache.
are you running rclone vfs/refresh before each plex scan?
fwiw, i do not use the vfs file cache, equivalent to --vfs-cache-mode=off
for the emby media scan, to minimize the amount of data downloaded,
i have configured emby to not create thumbnails, previews and so on.
otherwise, for each new media file, emby would have to download the entire media file.
Just to give you an idea, I have "Upgrade media analysis during maintenance" checked in Plex' Scheduled Tasks, and every night from 1AM to 8AM it analyzes roughly 5,000 items. I believe Plex counts one media file as three separate items for this purpose, so you do the math
Like @asdffdsa, I do not use the cache. A full library scan of ~250,000 media files takes about 4 minutes.