Most likely the reason it seems to slow down is because it starts out transferring some large efficient files and then hits a group of small ones. That will start to reduce your reported average speed.
If so then the only real workaround for that is to increase transfers further. as NCW said you should be able to go significantly higher than 10, and that will definitely help.
You can also further increase the chunk sizes if you have plenty of ram to spare. This won't help for the small files, but it will make the larger transfers even more efficient.
If none of this works - I think I would doublecheck that it isn't an actual network issue - that you actually have 100Mbit of useable bandwidth at your current location. Anything from traffic from other users to dodgy wifi can cause less than expected throughput. A simple speedtest.net check should suffice.
EDIT: You might also take a look at your data and see if you recognize any folders that have mostly large groups of small files. Archiving those types of folders before upload will make it far more efficient to transfer. This is more of a workaround than a solution - but in some cases it can absolutely be worth it. Any time a folder contains more than a few thousand files I generally ask myself it would make sense to archive this before I store it. Hopefully in the near future there will be an rclone backend that can handle this sort of thing automatically in the background.