Testing for new vfs cache mode features

Yeah, but if you have 2 files going at the same time and they over your cache size, it's going to be delete and re-add bits and to me feels like a waste of the cache if it dumps files in and out. The design for the cache is not chunk based, but file based so you'd really want some wiggle room depending on what you want to do.

Rclone only reads what is requested on the mount. You have an application that is doing a directory list based on that output as that's a standard directory listing.

With the last few beta's and now v1.53.0 I've been toying around with the new vfs cache.

This seems to work very well so far, however I've been trying to investigate some Plex videos taking between 1-2 mins to start.

Rclone settings:

rclone --config /config/rclone.conf mount --allow-other --cache-dir=/data/cache --drive-skip-gdocs --vfs-cache-mode=full --vfs-cache-max-age=300h --vfs-cache-max-size=500G --dir-cache-time=1000h --log-level DEBUG --log-file=/log/rclone.log --poll-interval=15s --user-agent=myagent --rc --rc-addr=:5577 gdrive-data: /mnt/netdata-testing

Now, for every couple of video-starts I've been timing, all looks ok in the logs as it fetches the ranges, etc. and starts playing typically within 3-10 seconds, however when this issue occurs, I typically have waited for around 30-seconds after hitting 'play' in Plex and I would get this in my rclone logs:

2020/09/02 22:06:05 DEBUG : download/tv/someshow/Season.01/someshow.-.s01e07.WEBDL-1080p.mkv: vfs cache: stopping download thread as it timed out

Once this is hit it typically takes another 30 seconds+ for (presumably the range to download) and for it to start playing. This however does not always appear at the start (with the 4x file opens) of a stream either - this will happen many times mid-show, and it appears the player catches up to the end of the buffer and then stalls, whilst in the logs you can see it getting the above error, then pulling data again and eventually (1-2mins later) start playing again.

This of course points to a connectivity issue but I'm 7 hops away and less than 5ms away at GigE to what I see rclone connecting to for Gdrive and this issue is not occurring when using !vfs-cache-mode-full

Keen to hear if others have experienced/seen this.

FWIW it seems to occur quite a bit -

$ grep -c 'download thread as it timed out' /docker/rclone-beta/log/rclone.log
4273

Thanks!

0 for me.

felix@gemini:/opt/rclone/logs$ grep -c 'download thread as it timed out' rclone.log
0

Today I installed the stable version on windows with the update of the full vfs cache for testing.
The disk was mounted, I have access to it but when analyzing the logs I saw several errors equal to the test when reading the files.

2020/09/02 23:15:38 ERROR: HoltzFlix / Tokusatsus / Super Sentai (1975) / Season 10 / Super Sentai (1975) - S10E14 - Love And Turn.mp4: vfs cache: failed to _ensure cache EOF

My montage: mount --poll-interval 1m --fast-list --tpslimit 10 --dir-cache-time 2m --vfs-cache-mode full --vfs-cache-poll-interval 1m --vfs-cache -max-size 30G --vfs-cache-max-age 72h --vfs-read-chunk-size 32M --drive-chunk-size 16M --cache-dir C:\rclone-cache --buffer-size 16M --log-file C:\rclone-logs\rclone.log --log-level NOTICE tdrive:/ g:

OS: Windows 10
Create Service: NSSM
Create Disk: WInFSP
SSD
8GB RAM

That's reported in the release thread already as well.

Probably makes sense to close this out since we're stable.

1 Like

This happens when no data has been read from the download thread for 5 seconds. It is a normal part of how the vfs cache works.

This is probably indicative that you need to use a bit of

  --vfs-read-ahead SizeSuffix              Extra read ahead over --buffer-size when using cache-mode full.

If --buffer-size isn't enough time for the reader to start up again without skipping. This buffers to disk so you could make it 512M and rclone would read up to 512M ahead of where you've got to.

You could also try increasing --buffer-size also (the default is 16M) if you made it 64M that wouldn't be excessive. Remember --buffer-size is buffered in RAM though, so don't make it too big!

Appreciated!

Yep that got it down in count at least 50x - now just trying to find the sweet spot between --buffer-size and --vfs-read-ahead

Thanks!

Are you direct playing or transcoding/direct streaming the items you are having issues with?

Ah to set it straight - my buffering-pauses went away with vfs-read-head (just it appears 50x less in logs) , though it appeared with both direct-playing and transcoding before. I try and test with direct-only to eliminate the transcode-layer usually though.

Direct playing makes sense if something was hung up. I direct play everything and have never seen a hung up so my curiosity is peaked.

Transcoding does not though as that uses the Plex read ahead buffering so it keeps 'x' ahead.

Is this cache LRU ? Does it work with union remotes just fine? If all union remotes has the identical data, can I swap them without invalidating the cache?

I don't know if this has been asked before. Willing to give a shot again now that this cache left beta

Are there any flags to control the vfs cache size now?
I tried using
--vfs-cache-max-size=20G --vfs-cache-max-age=10s --vfs-cache-cache-mode=writes
--vfs-cache-max-size=20G --vfs-cache-max-age=10s --vfs-cache-cache-mode=full

But it fill up my 120GB free space

The flags look correct. Please create a new thread and fill the template along with the debug logs so we can see what's wrong.


@ncw @Animosity022 Can we lock this thread now that the feature has been released in the stable builds?

I think most people like me will start really testing it now that is out of the beta.

Since I'm having 403 errors I'm open to investigate another route to mitigate that with a local cache

That should probably be a new stickied thread similar to this one. I am just interested in separating it because this was mostly dev testing with the various flags to decide on the defaults etc. and now that the feature is in stable builds, any testing should proceed with the defaults.

It would be nice indeed a new topic explaining the defaults and what they do in detail. like a FAQ I feel that would help a lot...

1 Like

There is quite a lot of help about the VFS here.

https://rclone.org/commands/rclone_mount/#vfs-virtual-file-system

It probably needs its own page!

I think closing this would be good and perhaps a new thread can get started up.

Ok I'll lock this thread now the VFS changes have made it to the release.

Please ask new questions in a new topic - thank you :smiley:

1 Like