Good!
It may be worth experimenting with --vfs-read-ahead
Also good!
Try increasing VFS read ahead, that may help.
I haven't experimented with performance yet. Also try without debugging, the debugs are excessive right now!
Sorry but how do we disable the debugging ?
And, the vfs read ahead parameters is working on windows ? I believed it was only doing something on linux.
EDIT : oh it's working all right. And no speed problem here, I've a 8gb fiber link here, and rclone pull data around 2-4gb/s. But it seems it's pulling a lot more than necessary. Like in a few minutes my vfs cache is already at 44go (size on disk), and no way my torrent program has uploaded that much.... or accessed that much data. Maybe my vfs read ahead is too high ... Here my rclone mount command :
rclone.exe mount --cache-dir c:\mountcache --attr-timeout 8700h -vv --stats 10s --buffer-size 16M --vfs-cache-poll-interval 30s --poll-interval 60s --dir-cache-time 2h --async-read=true --vfs-read-wait 5ms --vfs-write-wait 10s --vfs-cache-mode full --vfs-read-ahead 16M --vfs-read-chunk-size 16M --vfs-read-chunk-size-limit 1G --use-mmap --local-no-check-updated --drive-chunk-size=256M --multi-thread-streams=4 --multi-thread-cutoff 250M --transfers 8 --drive-disable-http2=true --fast-list --vfs-case-insensitive --vfs-cache-max-size 200G "gdriveencryptedownapi:seedold" G:
tested this morning with debug log. I Think when I download the file opens / close serval times. When I abort this download (browser download via nextcloud) the server keeps reading the file. Netdata shows me alot of IOWAIT.
Ok so here it's caching the whole files, even if a few blocks are required by the torrent client. I put vfs read ahead at 0M, same thing. Right now for 600mb read, I've 34gb written to the disk as caching...
All right so, Even with buffer at 0 and small chunck, etc, same behaviour. So, I produced a log file, too big for pastbin even if it's only about 2 minutes of time. During that time, around 8gb were downloaded and written to the disk, And around 80mb read from the disk where the cache mount is.
Before this log file, i let rhe mount run for about an hour, to let the cache "populate", and I was around a 15-18 ratio read / write to the disk, so still huge.
Here is the log file :
https://web.rootax.org/temp/debugrclone.zip
Hope it can help you to understand what is wrong (IF something is wrong) or me to understand why so much caching when the data read is pretty small.
Thx !
Just don't put -vv - use -v instead. Useful for testing, but if stuff goes wrong I'll needs a -vv log.
Ah, I think that is a problem I haven't addressed yet - if you've got a really fast internet connection that downloads much faster than the read app then quite likely rclone will end up downloading everything..
How did you measure size on disk? Rclone is using sparse files so it isn't that easy to measure the size on the disk any more!
The new --vfs-read-ahead parameter will, the one for the mount --max-read-ahead I can't remember whether that works on windows or not but it controls something different.
Ah...
Yes....
So I need to fix that I think. If the client leaves the file open then rclone will just read to the end of the file in the background. Great for caching, not so great for disk use!
What rclone needs to do is stay ahead of the read point by the -vfs-read-ahead parameter but not read more than that.
I'll have a think about how that might work...
For the size on disk, windows show the difference between "size", and "size on disk". I take than size on disk is the real data written on the disk. And, I've sort on confirmation thx to the program Primocache, which is also reporting the "size written" to the volume (in my case a dedicated volume for the rclone cache dir, so testing is easier). I monitor the data read with primocache too.
Aaaaaannnnd, i see crazy speed and write activity in the task manager, so with all that, my guess is what i described is pretty much what is happening, in my case.
Yes that sounds right...
I'll work on a fix - I have a cunning plan! I knew the current arrangement was too simplistic and would have to be fixed at some point....
2 Likes
Thx a lot : ) If I can help with tests and logfiles, let me know.
EDIT : I put the cache to writes for now. Maybe the "bug" is still present and downloading more than needed in this mode, but at least it wont write on the disk.
But, even in writes mode, I don't see a slowdown like Calisro.
Just a few questions.
Can this be made configurable? This would be useful in the case of multiple long streams happening from the same mount with a small total chunk-size where the oldest cached chunk is not needed but the oldest cached file is still being streamed.
Looking at the logs, it seems to be downloading in chunks of 32 KB? Is that configurable? This is mostly for performance reasons where many servers have multi-gigabyte connections to download with.
Is there any way this can be compatible with the existing read-chunk-size and buffer-size flags for specifying the chunk-size and the read-ahead to simplify the mount?
If the above is not possible, how do the existing read-chunk-size and the buffer-size affect the caching or is that a completely separate system?
Btw, the current implementation looks and works brilliantly, as expected. Kudos on that.
I tried different variations of read ahead (with debugging off) and performance is still very slow. (I'm bypassing the reverse proxy for testing as well). Im seeing that slowdown using 'FULL'. Writes is working as expected after the last beta.
Thanks, I will! I'll need a few days to come up with the next iteration I think.
The storage is done with sparse files rather than chunks. This simplifies the internal logic enormously and means there is always something on disk which you could read straight away. Unfortunately it means that evicting chunks is much harder - it is possible you can delete chunks out of sparse files - that isn't in my roadmap for the moment though.
Hmm 32k is the default copy size for io.Copy and I think that is probably where that limitation is coming from. That could and should be made bigger I think, probably to whatever --buffer-size is set to.
Yes I'd like to re-use existing flags where possible! --buffer-size is a good one. The --read-ahead one is a mount flag wheras I need one for the vfs layer... I could deprecate the one in the mount and use the one in the vfs layer instead. They do subtly different things and the one in the mount doesn't always work...
I don't think the current cache mode full code uses this flag yet. It probably should!
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks. (default 128M)
The --buffer-size will be used as a read-ahead buffer for the download as works currently.
Thanks 
I took the time to untangle the VFS code and separate out the caching layer from theVFS layer. That makes it so much easier to code on now, but it was a lot of pain doing it!
The read mode full code does quite a bit of copying at the moment
- mount requests data
- vfs downloader fetches it to disk
- read code reads it back from disk
- read code returns the block to disk
That could do with a bit of optimisation!
I suspect it might just be the 32k chunk size that is causing the problem though as identified by @darthShadow
3 Likes
The good news is it seems stable for now. I stress it for hours, zero lockup or anything. I get there are optimisations and bug fixes to make, but for a first public vfs read caching release, it's very good. Very good job imho.
1 Like
I believe that should be the --vfs-read-chunk-size flag instead? And possibly have the same behaviour as existing, i.e. doubling it till it reaches the --vfs-read-chunk-size-limit
--buffer-size would be better translated to the new vfs read-ahead.
I think the mount read-ahead can be left alone for now, if as you say they perform different things. I mostly meant translating the existing flags to the new flags:
vfs-read-chunk-size -> chunk-size for downloads
buffer-size -> vfs-read-ahead
2 Likes
In case this helps convince you to add it to the roadmap 
, both Windows & Linux do support zeroing out chunks to reclaim space, but it seems to be limited in the filesystems that it supports in Linux, however it does support all the popular ones. Docs:
1 Like
To make this more noticeable, I pinned it so folks should see it more often now as it'll stay at the top. It'll expire at the end of the month currently.
4 Likes
Hey Nick,
It looks like you are storing meta by way of Item entries in go. Didn't see how this translated to disk storage though. Are you going to make the vfs meta (chunks and/or internal db) available to be stored on separate a storage medium? --vfs-cache-db /nvme --vfs-cache-files /hdd
Also, will you be allowing parallel uploads? --vfs-cache-upload-limit 99
Also, will you be allowing a separate upload bw limit? --vfs-cache-upload-bwlimit 99M
Here is the next beta
This should hopefully sort out downloading issues like downloading the entire universe in the background!
https://beta.rclone.org/branch/v1.52.0-044-g11d5be41-vfs-beta/ (uploaded in 15-30 mins)
- Download to multiple places at once in the stream
- Restart as necessary
- Timeout unused downloaders
- Close reader if too much data skipped
- Only use one file handle as use item for writing
- Implement --vfs-read-chunk-size and --vfs-read-chunk-limit
- fix deadlock between asyncbuffer and vfs cache downloader
- fix display of stream abandoned error which should be ignored
3 Likes
I'm storing them on disk in a parallel directory structure as JSON blobs, so in ~/.cache/rclone/vfs you'll find the data and ~/.cache/rclone/vfsMeta you'll find the metadata.
You can use --cache-dir to direct where you want the cache to go. There isn't a separate setting for the metadata - it is very small though.
The writeback cache allows --transfers uploads at once.
I think that is a separate issue (I'm pretty sure there is one) for a --bwlimit and a --txbwlimit and --rxbwlimit flags. Or make the bwlimit flag take some more funky parameters.