I am currently working on a high-performance media server project ("Highlander") that utilises Nzbdav WebDAV mounted as a backend. My library consists of ~217k media files (symlinks) and a significantly larger metadata database.
I have successfully implemented a RAM-based Metadata Cache and a 32GB RAM pool to protect my SSDs from wear, but I’ve run into a bottleneck during the warming aka "priming" of the directory cache.
The Use Case:
Since our WebDAV provider is local (nzbdav runs in a docker) and can handle high concurrency without API rate-limiting, we want to maximize the speed of the initial vfs/refresh. Currently, the vfs/refresh command via the RC interface appears to be single-threaded (or limited in its recursion speed), making the initial "walk" of 217k files take significantly longer than the hardware is capable of.
The Proposal:
I would like to propose adding support for a threads=X parameter (or exposing --vfs-refresh-threads) to the vfs/refresh Remote Control command.
Allowing the recursive walk to happen in parallel would:
- Drastically reduce "blind" time for media applications (Plex/Arrs) after a mount restart.
- Allow power users with high-thread-count CPUs (Threadripper, etc.) to saturate their local APIs/fast internet connections during the priming phase.
- Improve the "System Health" in environments where a massive directory cache TTL (we use 200d) is required to maintain performance.
I have tested various dir="" and recursive=true combinations, but the inability to specify parallelism in the refresh call remains the primary bottleneck for large-scale local WebDAV implementations.