Yeah, but if you have 2 files going at the same time and they over your cache size, it's going to be delete and re-add bits and to me feels like a waste of the cache if it dumps files in and out. The design for the cache is not chunk based, but file based so you'd really want some wiggle room depending on what you want to do.
Rclone only reads what is requested on the mount. You have an application that is doing a directory list based on that output as that's a standard directory listing.