and, is it normal then that I'm getting such a behavior when browsing my vfsMeta directory ?
as you can see, I can open any folder in there, and all of them are empty. Should I not see the names of my actual files?
You didn't click on the link I shared and read it.
--vfs-cache-mode full
In this mode all reads and writes are buffered to and from disk. When data is read from the remote this is buffered to disk as well.
In this mode the files in the cache will be sparse files and rclone will keep track of which bits of the files it has downloaded.
So if an application only reads the starts of each file, then rclone will only buffer the start of the file. These files will appear to be their full size in the cache, but they will be sparse files with only the data that has been downloaded present in them.
This mode should support all normal file system operations and is otherwise identical to --vfs-cache-mode writes.
When reading a file rclone will read --buffer-size plus --vfs-read-ahead bytes ahead. The --buffer-size is buffered in memory whereas the --vfs-read-ahead is buffered on disk.
When using this mode it is recommended that --buffer-size is not set too large and --vfs-read-ahead is set large if required.
IMPORTANT not all file systems support sparse files. In particular FAT/exFAT do not. Rclone will perform very badly if the cache directory is on a filesystem which doesn't support sparse files and it will log an ERROR message if one is detected.
One thing I'd like to point out is that with a writeable mount you do not want to use --no-checksum and --poll-interval 0. Otherwise checksums will never get compared and the mount won't update on file changes. I use those with my read-only mount, because nothing ever changes within that mount itself. Does that make sense?
It actually does. No poll interval = rclone not checking for changes.
here is my current script content :
rclone mount --attr-timeout 5000h --dir-cache-time 5000h --drive-pacer-burst 200 --drive-pacer-min-sleep 10ms --rc --vfs-cache-mode full --vfs-read-chunk-size 1M --vfs-read-chunk-size-limit off --vfs-cache-max-size 500G --vfs-cache-max-age 5000h --cache-dir F:/cache/ gcrypted:music_library/ X:
timeout /t 60 /nobreak
rclone rc vfs/refresh recursive=true --drive-pacer-burst 200 --drive-pacer-min-sleep 10ms --timeout 30m --user-agent
Roon is still scanning files while I write these lines. I have no clue if the newly added vfs/refresh line I added will improve things. I'll keep you posted!
Hi gents,
I wanted to comment on this after enabling this vfs/resfresh thing for a weeks or two.
I noticed that since I activated it, I indeed DO have files listed in my cache directory, under the "vfsMeta" folder, that previously was existing, but only listed the folders, and not the files.
Everything is indeed way faster now.
In short, I would recommend using vfs/refresh to anyone willing to handle and list a lot of small files on their mounted gdrive.
Thanks again for your help on this @VBB and @Animosity022 !
2 Likes
Guys, I have one extra question and I can't figure it out on my end.
Why is the vfs/refresh erasing everything at server reboot ?
I explain myself :
-> once refresh command is run, and when files are "scanned" I can clearly see in e:/cache/vfsMeta/Folders/FILES
-> once the computer gets restarted for any reason, I end up only keeping e:/cache/vfsMeta/Folders
= the file list is gone, but the folder list remains.
Is there any way to make the file list permanent ? I have a fixed folder content (and I only do update a monthly folder, then add a new one, etc. So can I tell rclone to keep the file cache of all folders, except the last monthly folder, permanent ? This would save me a LOT of time when Roon is rescanning stuff.
Another option would be to tell rclone to scan for all the files all alone, without the need of an external app (Eg: Roon in my setup) to request for them to be listed.
Any great idea ?
Thanks again ! 
hi,
the rclone vfs dir cache is not persistent across reboots.
run rclone rc vfs/refresh as a task on reboot
and/or
run rclone rc vfs/refresh on demand, before having roon do its scan.
maybe @VBB can comment?
1 Like
I do this refresh once a day, right before I run a Plex scan. Otherwise the dir cache (not to be confused with an actual Rclone cache) would be stale.
Since I don't use the real cache, I don't know exactly how it behaves, but I would think whatever is cached in there (files and folders) would survive a reboot, no?
hi,
the real vfs cache does survive reboots.
as i learnt from you, i no longer use vfs cache when streaming media with emby.
but then, we both have very good internet connections.
Right, but what @silkyclouds is saying here sounds like his actual cache is not retaining the files after a reboot.
-> once refresh command is run, and when files are "scanned" I can clearly see in e:/cache/vfsMeta/Folders/FILES
-> once the computer gets restarted for any reason, I end up only keeping e:/cache/vfsMeta/Folders
= the file list is gone, but the folder list remains.
We know that the dir cache (the one that the refresh command affects) only exists in memory.
hmm, too two many caches. jojo no know understand, need --magic
tl;dr, the OP needs to run a test, reproduce the problem and post the debug logs to prove it.
the OP seems to be using multiple locations for the vfs cache, perhaps he/she/it is getting confused, like jojo?
--cache-dir F:/cache and e:/cache or perhaps that is a simple typo
the actual/real vfs cache can survive between each execution of rclone mount, a reboot should not change that.
keep in mind that files, once read from but no longer accessed, will stay in the vfs cache for a period of time defined by
--vfs-cache-max-age duration Max age of objects in the cache (default 1h0m0s)
this can be seen in the debug log.
let's say i start a rclone mount, watch a media file, kill the mount, restart the mount, that media file will still be in the vfs cache.
at the start of the mount and once per time period defined by
--vfs-cache-poll-interval duration Interval to poll the cache for stale objects (default 1m0s)
rclone will scan for and prune out stale files.
DEBUG : vfs cache RemoveNotInUse (maxAge=3600000000000, emptyOnly=false): item folder/file.mkv not removed, freed 0 bytes
1 Like
Hi,
let me clarify things. You speak about the actual file cache. The files that gets downloaded, and end up on a local drive.
regarding E: and F/, it indeed is a typo 
now lets dig this.
Files gets downloaded locally, and are cached in e:/cache/vfs
there, you can list the files that were read by Roon, and only the needed chunks are stored on the local drive.
there is one other thing (please read above posts) which is launched in the same batch file as the actual rclone mount. It is this command :
rclone rc vfs/refresh recursive=true --drive-pacer-burst 200 --drive-pacer-min-sleep 10ms --timeout 30m --user-agent
This command seems to store the file listing, creating file list under e:/cache/vfsMeta/
the "files" inside e:/cache/vfsMeta/ are empty, and connot be read. They are there ONLY to allow a faster listing (and in my setup, scanning) of the files by the apps that calls them.
What I am saying, is that the file structure which is "cached" by the second command I listed here (this is totally independent from the rclone mount, vfs cache, cache max timeout etc.) disapears once my computer reboots.
As you guys introduced me to vfs/refresh, I was hoping you could share its behavior after a reboot. And yes, @VBB , I am also running this at startup, as I simply added the vfs/refresh command right after the mount, in the same batch file.
Thanks !
No, it doesn't. That command is only for dir-cache and that's in memory.
That's not rclone doing it as it's persistent on reboots. You'd want to generate a log file and reproduce the issue and you can see it's not rclone doing it.
that is not correct and i understand the confusion.
to be clear, there are two caches.
each cache has:
a. different purpose
b. different data
c. different storage
d. different levels of volatility
-
dir-cache
a. to speed the listing of folder/file names from slow cloud based storage.
b. a list of folder/file names.
c. stored in memory.
d. does not survive a reboot. -
vfs-cache
a. to store the contents of files read from slow cloud storage. suffice explanation for now.
b. the contents of files.
c. stored in local storage, in your casee:/cache.
d. does survive a reboot.
How is your file structure ?
If you want to speed up things, for optimal filesystem and listing performance I usually do not have than more 30 files/subfolders in each folder.
You could try something like this for music:
/YEAR(OF RELEASE)/MONTH/DAY/
If that's still not enough (if you are as extreme as me) just use random numbers /XX/XX/XX/XX as much you need to keep each folder with a maximum of 30 objects in it.
You may think it doesn't but this make massive. I have one folder like this with 5k objects and it takes rclone ls fast list 3.4s to list it all, while in 5s of another folder not like this it can't list A SINGLE object in 5s!
I indeed do create a file structure based on monthly folders, and 30/31 daily folders inside the monthly folder. Your observation is correct. And I was already using this folder convention method
wait, it means I should mount my rclone using both vfs cache AND dir cache ?
and what is the vfs/refresh option then doing ? I am quite confused, people here tend to tell me the vfs/refresh is actually keeping the file structure in memory ? right ?
rclone rc vfs/refresh, that is for the vfs dir cache, the list of directory/file names stored in memory.
it will pre-cache all the directory/file names in memory.
for now, just use your last posted command as is, with both caches.
-
the vfs dir-cache is always enabled.
-
the vfs file cache is optional.
for example, for my emby media server, i do not use the vfs file cache.
in effect--vfs-cache-mode=offso when i play a media file, rclone will stream it direct from the cloud, not caching it on local storage.
let's say i have a movie, 1GB in size and i watch it twice in a row.
-
--vfs-cache-mode=off
rclone will download it twice, 2GB downloaded -
--vfs-cache-mode=full
the first time i watch it rclone will save the file in the vfs file cache, on local storage.
the second time i watch it, rclone will already have it in the vfs file cache and will not download it again.
so rclone will have only downloaded 1GB.
1 Like
Regarding the amount of sub-folders/files within a folder, this becomes moot with the dir cache. I used to think that my extremely slow Plex scans were directly related to how Windows Explorer handles the metadata (or initial lack thereof), but after switching to the new Plex scanner this hasn't been an issue at all. I went from 2.5 hours of scan time for a folder with 13,000 sub-folders to now less than five minutes. So, for Plex use, it's perfectly fine to have lots of files and/or sub-folders, as long as you refresh the dir cache before a scan (and you're using the new scanner).
1 Like
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.