Most TV apps has as super tiny buffer and would notice any blips in a constant connection. I tend to avoid all the TV built in Apps and would use a Shield/Roku 4K/ATV 4K.
Latency is one part of the equation, but any packet loss would also come into play. For some perspective, I have a 8ms ping to my resolved Google Drive API endpoint. Are you suing a US Google Drive or a European one? Anything to reduce latency down would help.
I thought you removed the cache? It's more a juggling act based on your specific setup. I have my setup documented that works very well for my very specific use case and I try to share what works for me. The problem with any guides is that your setup is so different from mine, it makes it tough to say exactly what would work for you. Some folks have had great success with using the cache backend. Some folks have had great success using plexdrive instead of rclone. Use what works for you and if something doesn't work, you have to experiment a bit. I used plexdrive for ages because it worked better. rclone works better for me now so I use that.
If it's direct streaming, you should be able to make this like 300 seconds or even higher. This means the transcoder will buffer up 5 minutes for me:
This does nothing when you are direct playing though as it's not using the transcoder, which is why I suggested making a bigger --buffer-size as that may help out as that means rclone will read ahead and keep that much in memory. The problem still means the VPS has to talk to the plex player so if there's latency / packet loss from that connection, nothing will help as if the buffer empties on the player, you see the buffering message.