Yeah, but if you have 2 files going at the same time and they over your cache size, it's going to be delete and re-add bits and to me feels like a waste of the cache if it dumps files in and out. The design for the cache is not chunk based, but file based so you'd really want some wiggle room depending on what you want to do.
Rclone only reads what is requested on the mount. You have an application that is doing a directory list based on that output as that's a standard directory listing.
Now, for every couple of video-starts I've been timing, all looks ok in the logs as it fetches the ranges, etc. and starts playing typically within 3-10 seconds, however when this issue occurs, I typically have waited for around 30-seconds after hitting 'play' in Plex and I would get this in my rclone logs:
2020/09/02 22:06:05 DEBUG : download/tv/someshow/Season.01/someshow.-.s01e07.WEBDL-1080p.mkv: vfs cache: stopping download thread as it timed out
Once this is hit it typically takes another 30 seconds+ for (presumably the range to download) and for it to start playing. This however does not always appear at the start (with the 4x file opens) of a stream either - this will happen many times mid-show, and it appears the player catches up to the end of the buffer and then stalls, whilst in the logs you can see it getting the above error, then pulling data again and eventually (1-2mins later) start playing again.
This of course points to a connectivity issue but I'm 7 hops away and less than 5ms away at GigE to what I see rclone connecting to for Gdrive and this issue is not occurring when using !vfs-cache-mode-full
Keen to hear if others have experienced/seen this.
FWIW it seems to occur quite a bit -
$ grep -c 'download thread as it timed out' /docker/rclone-beta/log/rclone.log
4273
Today I installed the stable version on windows with the update of the full vfs cache for testing.
The disk was mounted, I have access to it but when analyzing the logs I saw several errors equal to the test when reading the files.
2020/09/02 23:15:38 ERROR: HoltzFlix / Tokusatsus / Super Sentai (1975) / Season 10 / Super Sentai (1975) - S10E14 - Love And Turn.mp4: vfs cache: failed to _ensure cache EOF
This happens when no data has been read from the download thread for 5 seconds. It is a normal part of how the vfs cache works.
This is probably indicative that you need to use a bit of
--vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full.
If --buffer-size isn't enough time for the reader to start up again without skipping. This buffers to disk so you could make it 512M and rclone would read up to 512M ahead of where you've got to.
You could also try increasing --buffer-size also (the default is 16M) if you made it 64M that wouldn't be excessive. Remember --buffer-size is buffered in RAM though, so don't make it too big!
Ah to set it straight - my buffering-pauses went away with vfs-read-head (just it appears 50x less in logs) , though it appeared with both direct-playing and transcoding before. I try and test with direct-only to eliminate the transcode-layer usually though.
Is this cache LRU ? Does it work with union remotes just fine? If all union remotes has the identical data, can I swap them without invalidating the cache?
I don't know if this has been asked before. Willing to give a shot again now that this cache left beta
Are there any flags to control the vfs cache size now?
I tried using --vfs-cache-max-size=20G --vfs-cache-max-age=10s --vfs-cache-cache-mode=writes --vfs-cache-max-size=20G --vfs-cache-max-age=10s --vfs-cache-cache-mode=full
That should probably be a new stickied thread similar to this one. I am just interested in separating it because this was mostly dev testing with the various flags to decide on the defaults etc. and now that the feature is in stable builds, any testing should proceed with the defaults.