Thanks for the advise @asdffdsa
I'm freeing up some space on my main HDD to set the cache there and if I'm still facing the issue I will start like that.
Edit;
This is very odd.... After doing the above the issue was still there so I've done another test and realised that the issue is just with the mount...
If I do the following I pass the 10M mark no problem....
rclone copy -P "remote:Movie.mkv" ~/
Transferred: 1.506G / 2.332 GBytes, 65%, 36.065 MBytes/s, ETA 23s
I give up for today