Idea of a new Mode called "vfs-cache-mode light"

Hi,

i have an Idea of a new Mode let's call it "--vfs-cache-mode light"

What should this Mode do? It should pre-download small parts / chunks of the files on a mount.

Why? Other Modes wörks great for bigger files, but not for small files like Music files wich Need gapeless Playback between two files, there are loading to slow from Cloud (gdrive).

So my idea, why not preloading the first parts of the files. If we browse on a dir on a mount simple preload some small parts of the file instead of downloading the whole file in mode full.

It works with a Cache remote, but whould be great if it will works on vfs-cache too.

Thx for reading.

The whole VFS backend is being rewritten to incorporate more of the cache backend.

You'd think with a global lockdown @ncw would be done already with it but he still needs more time :slight_smile:

Just keep checking back as I know he's getting closer to releasing it in beta form and it somewhat merges a lot of the cache functionality into stock VFS.

1 Like

:laughing:

You've described pretty much what the reworked vfs code will do! It will download parts of files and cache them on disk for you. You'll need --vfs-cache-mode full for this.

So let's say you have plex/whatever doing media scans. You have a 5 GB video file. Plex reads the start and the end to work out what kind of file it is and generate a thumbnail (say - I don't really know what plex does!). The new cache mode will just download the parts of the file that plex reads and store those on disk in a sparse file. This file will look like it is 5 GB in size but it will only actually have the data plex has read in it and so will only take a few MB of disk space. There will be another small metadata file recording what data rclone has read so it knows it doesn't need to read that again unless the upstream object changes.

I think when all the wrinkles are worked out, --vfs-cache-mode full should be the best cache mode to use. It won't download the whole file before starting (like it does now) and it will deal really well with seeking, random writes etc.

Anyway I have it basically working but I'm concentrating on getting 1.52 out the door before I put it out for beta.

7 Likes

Hi, sounds really nice.

Can't wait to test this :slight_smile:

Thanks for your answer.

2 Likes

Looking forward to it! :slight_smile:
What you and the team supporting you have done is amazing.

Rclone works flawlessly and still keeps improving, thank you guys!

2 Likes

So 1.52 was released (thanks for it), any timeframe for the beta with the new vfs Cache System?

Thank you.

I'll push up a branch soon for testing :smiley:

5 Likes

Would be nice to be able to store the sparse files on ram. And a way to control how much ram the cache would use, by chunk sizes maybe?

If I'm reading at 1 Gbps from the mount, and using full cache mode, is there a way to calculate how much disk space / ram will be needed?

Looking forward to try your new code!

You'll be able to set total size of the cache.

I think disk space will be the largest file you want to stream probably

1 Like

Can't wait (lot of seeding from the cloud here) !

1 Like

I'am Right that "https://beta.rclone.org/branch/v1.52.0-027-g799e77f4-vfs-beta/" is a beta of the new VFS Cache feature?

i have to compile it by myself but it's not working for me, i can only open some smaller file, but no bigger files like > 2MB.

Btw: Why are Linux and Windows binarys are missing?

Tested on Debian 10 & Windows 10.
mount Settings are very Basic like: rclone mount Google-G-Suite: N: --vfs-cache-mode full -vv

Here is a log from the issue: https://pastebin.com/0VSzSPQe

give @ncw some time. I think he will inform us when we can start testing.

1 Like

It'll be announced properly when he's ready to test so if you think about it, the more he has to respond to when is it ready is time he could be coding it :slight_smile:

1 Like

Yes.

Interesting! What is the symptom when you open a bigger file? I couldn't see an error in your log.

That was the first time I ran it through the CI in a while and there is stuff I need to fix!

Hi,

On Windows Nothing happens at Clicking to open a File (Movie, Music) and on Linux the Mediaplayer is Opening, but will Freeze and Nothing happens.

I just checkout out the vfs branch on GitHub and compile it, i don't think there are to much Things that can be wrong?

I expect it is a bug - I'll investigate!

I don't think you will have compiled it wrong.

Just to inform you.
Still no changes with the latest vfs changes.

After Clicking that mp3 file file Explorer freezes an Nothing happens.

Here a log: https://pastebin.com/1zRDah9r

I'm concentrating on getting the CI to build clean. Made huge progress today! But more to do.... I'll look at your bug once I've got the CI reliable.

I've fixed the CI and I've also fixed the deadlock I think you were seeing @root0r

The tests pass reliably now and it worked in some light testing.

Here is a new beta if anyone enjoys living dangerously and wants to test --vfs-cache-mode full

https://beta.rclone.org/branch/v1.52.0-037-g2a7af410-vfs-beta/ (uploaded in 15-30 mins)

Hmm Maybe I should re-arrange the downloads so the branch name is its own directory...

If it locks up again can you send a full backtrace, so kill the mount process with kill -QUIT pid and send the output - thanks.

1 Like

Hi,

i will give it a try now and post Feedback asap :slight_smile:

Thanks for your support.

1 Like