as you might know, rclone has two caches.
one cache, vfs file cache, to store chunks of files.
one cache, vfs dir cache, to store the dir/file names
this might be a helpful summary
https://forum.rclone.org/t/status-about-using-rclone-for-music-storage-playback-in-2021-access-times-improved/27648/34
you can prime or pre-load the vfs dir cache, which can be useful, before a media server scan.
rclone rc vfs/refresh recursive=true
so if you do a ls -R on the mount, should be much faster.