welcome to the forum,
that is how rclone works.
check out my summary of the two rclone vfs caches.
https://forum.rclone.org/t/status-about-using-rclone-for-music-storage-playback-in-2021-access-times-improved/27648/34
might try --vfs-refresh, --network-mode and increase value for --dir-cache-time
with windoz explorer, disable detail pane, preview pane, thumbnails, etc..