A rough calculation on the 4GB say that it corresponds to holding 2-4M directory entries in memory, that is 1-2M from each remote. This sounds (much) too high, your worst case should be less then 87k + 4*52k = 295k objects, that should fit within 1GB. So something seems wrong with the memory usage - even if the Go garbage collector is a little behind.
I find it a bit disturbing and would probably try a memory profiling to find the explanation/issue, if it was something I relied on to run my business. The instructions to collect the profile are in this post. Up to you.
I don't know. The default --buffer-size is 16M, so I would think something in the range of 20-40M per --transfer (using default settings). I guess independent from the file size. Nothing compared to the 4GB.