I mean, lots of people have intelligent answers but I am gonna list my stupid reasons why I could not play 4k from rclone and realised the dumb little things I did :
Good luck 
... in case you are interested and for all other people who assume that rclone is making trouble in this regard: currently, I'm actually doing network debugging regarding my ISP as well as the provider of my dedicated server. Traceroute revealed a dramatic routing issue regarding the provider of my server. Ironically, they state that it comes from my ISP and my ISP states that the problem is located in the server provider's networks (what it is also shown in the traceroute data). Currently, the server provider is "working on it" 
... i really fear a never ending story.
@Animosity022
Please, let me ask you something because there might be something what might be solved by changing some mount settings...
Could you please have a look on the following Screenshot showing results from "vnstat" (command line tool to visualize the current traffic per second):
Left hand: received data = rx
Right: transmitted data = tx
Movie details:
The movie starts very fast and there is no buffering until there is a short but significant drop of data bandwidth (see red arrow) around 96 mb/s (coming from 130mb/s). Here the first buffering started/arose.
Furthermore, you see that the server is receiving data from my Gdrive (left column=rx).
These facts made me reconsider the current issue.
Looking at aforementioned vnstat output one may suggest a server side issue.
The server seems to gather data and instantly transfer it to the client instead of transmitting data from the cache (--vfs-cache-mode full is set!)... To me, it seems that rclone caches not enough data locally or is not fast enough to transmit more data. In this case I should be able to reconfigure the mount settings. The server has a 1GBit uplink what should be more than sufficient.
So, lets say rclone has more data cached locally (server), I would expect that the transmitted data / bandwidth is higher and/or even more stable.
Any ideas? Maybe there is something missing or odd in my mount settings (i reposted it below).
Please note that I have plenty of RAM (128GB) and space (see specs in my first post). There should be no problem using large buffer and cache sizes.
Furthermore, there is no need for a very qick start of a movie... I can wait for some seconds (currently it starts withing 3 sec).
Mount:
--rc \
--log-file ${LOGS}/plexmedia_rclone.log \
--log-level INFO \
--umask 002 \
--allow-other \
--dir-cache-time=160h \
--drive-chunk-size=512M \
--vfs-cache-max-age=72h \
--vfs-cache-mode full \
--cache-dir ${UPLOADS} \
--config ${RCLONEHOME}/rclone.conf \There's probably not much to do as it looks like the connectivity between your client - server might have a blip as the player doesn't have a large buffer on it and is sensitive to dips or not consistent bandwidth.
I've been testing a shield lately as I want to replace my 4K ATV and have had no issues throwing anything at it with very high bitrates and types of audio. My server/player are on the same LAN though so that removes any internet issues with the server-client.
@Animosity022
Sorry for bumping this thread but after problems got more and more, I got an idea for this and wanted to ask you for your opinion.
Unfortunately, I just don't get your actual setup (local server directly streaming from your Gdrive vs remote server at a data center what is equal to my current setup).
What if I use a local (at my home) server instead of at a remote server at a data center to reduce one source of problems? This local server would work exactly in the same way as my current remote server does (using rclone to stream) at the data center. The only difference would be that it is located at my home (and - of course - higher electricity bill 
).
My server and client are both in my house so I have only have 1 variable, which is my server connection to my GDrive, which rclone handles any minor hiccups or second or two blip in a connection.
As you stated a bit further down, if the server is remote, you have another variable that's outside of your control to watch/see/measure.
If my server was in another data center. I have a longer flow and it goes from client location -> server location -> rclone is going to Google API.
You'd have a heard time measuring and ensure that client->server is not having an issue since you don't control the server and can't "see" everything or know exactly how the bandwidth limiting works.
I think many people do use VPSs out there to serve and perhaps if they read these they can add feedback as I think some providers work well, but some do not. It also depends on how good the peering is on the Internet from you -> client and from your provider -> Google's API for Drive.
I use a UPS at home and and read my power consumption and the switch from a lot of spinning disks to GDrive was definitely a cost savings for me.