Well, that's why I said outside of Plex, meaning without going through the Plex server at all. Are you able to mount rclone directly on your home PC?
My setup is similar to yours, where I use a dedicated server to run Plex and upload to Drive. On my home PC, I also mount rclone, but I use MPC-HC for playback. I personally don't use Plex, but my friends and family do. I have 500/500 at the house, and absolutely no issues streaming even the largest files. I don't even use cache.