A short primer on pre-caching (or pre-listing) technique

@Monarch_de_Boulogne requested that I share some of the information I gave him via PM so that it might benefit other users. At some point I will try to aggregate some more of this information and practical guides into some sort of megathread and or wiki article, but for now I will just leave this here for those who are interested.

If anyone has followup questions on this topic I will also take the time to reply to this here. Maybe I can use this as a reference in the future for others who are curious about the topic...

2 Likes

One thing I forgot to mention above is that these two flags wil also be required for the mount command:

--attr-timeout 8760h
--dir-cache-time 8760h

Usually these values are quite low, invalidating the cache after a short time. These values are low by default because not all backends have polling, but on a backend with polling we can keep the cache data indefinitely and rely on updates to keep the information fresh instead of re-gathering the full listing every time we need them.
The specifics numbers here are just an example - I've just set them to be a year to effectively "disable" them. Without setting these timers to a high value it won't make much sense to pre-list as we will just evict most of that information from cache very soon thereafter (1 second default for attr and I think 5 min default for dir).

attr refers to caching file-attributes (size, modtime, name and createdtime for most backends).
dir refers to caching the directory information/structure.

@Monarch_de_Boulogne This information is relevant to you as I've forgotten to tell you this before, sorry :slight_smile:

I am trying to follow your instructions, please correct me if something is wrong or not ideal:

Firstly, start the mount with the RC:
(NB: Windows Powershell)

.\rclone mount cryptomator: Y: -vv --rc --attr-timeout 8760h --dir-cache-time 8760h --rc-user=time --rc-pass=xxx --rc-web-gui --fast-list

then, open a new shell and trigger the remote control:

.\rclone rc vfs/refresh --user time --pass xxx

is that more or less the idea?

How do I know it's working? I can't really tell unfortunately.

I think it the answer from the RC may be silent unless you add -v to the remote control line. But then it should answer "OK" when the operation is complete.

I would just recommend you just wait and get those scripts I promised you here:

That should be ready-made to work in a fairly intuitive way.
I know I am late on this, but I have not forgotten. I've had to deal with some health issues the last days. Starting to feel a little better now though. I will make an effort to get it done during tomorrow - and if for some reason I am unable to do the things I wanted I'll just dump you the scripts I use as-is. They work fine already but I wanted to clean up some rough edges so they are easier to use on any system - not just my spesific setup.

no worries, it's not urgent, i just got curious :slight_smile:
Please don't force it in regards to your health – take your time and get well soon!

I agree it's better to have the "nice" version, even if it means waiting some extra days or weeks. I know all of you guys are doing it in your spare time and I appreciate it.