Doesn't seem you read my post above?
This is how buffer-size works -> rclone mount
--fast-list doesn't do anything on a mount so it can be removed.
vfs-read-chunk-size doesn't have anything to do with memory. It makes a request to the remote for a range of data and starts to download it via that size.
direct_io breaks a few things, but if you aren't storing metadata on there or torrenting from it, you might be ok. I keep it off myself in my mergefs config.