I have been using samba to stream and like you said I totally agree with your assessment of network delay.
But it was has been dead stable for me even streaming from rclone, with never a dropped connection, but because of how there is no buffer nothing is smooth.
For me it wasn't buffering as I simply hadn't gone to that stage yet.
mergerFS recursing the mount was taking over 70s instead of 6s it normally takes, and the entire mount slowed to a crawl. I could see in htop that it would wait on IO (D) very frequently.
Apart from that, systemd behaved weird with mounts, so I just wanted to ditch this ancient protocol.
not sure your use case but running rclone vfs/refresh will pre-cache the dir cache.
for example, if i add some media files to my mount and now i need emby to scan that.
i run that command first and then have emby rescan the library.
I enabled that bit. My use case for this mount is simple. *arr sends stuff to the remote client, imports it (moves server-side), then a script moves it from the server to the cloud and behind the scenes it simply changes mounts but since all of them are in a single mergerFS mount, it makes no difference to Plex.