So I have an inelegant workaround that seems to be functional so far, but it's far from ideal.
Back in the dark, dark days of pre rclone 1.53, multithreaded downloads worked with cache-mode writes and above. Unfortunately this means the whole file downloads locally before you start playing it which is why it was scrapped according to this post "rclone mount" does not work with multi thread downloading using "--vfs-cache-mode full" with version v1.55.1 and v1.54.1 - #2 by ncw.
Being multithreaded drastically reduces your chances of getting connected to one of the slow IPs at google's end (at least for Australia users) but the trade off is that you have to use rclone 1.52.3 and wait for the file to fully download before it plays.
ncw has plans to bring this functionality back in a future version of rclone - Support multi-threaded downloads when downloading a file to the cache · Issue #4760 · rclone/rclone · GitHub
I guess we'll either have to hope google fixes their shit or we beg/implore/pay/threaten ncw to prioritize re-adding this feature as quickly as possible 