I had a docker image running openVPN but I set it to use host network instead of bridge network. This affected the host network speed as it was going through the VPN too. Hope it help!
I am now getting close to windows speed with speedtest on the PI4 and having around 30-40 MB transfer speed using rclone.