I think the issue is indeed the Linux kernel that I am using, 5.14 rc6. Now if this is a kernel regression or just unstable, have no idea. However there are the performance numbers:
Kernel 5.14rc6 source remote, target local => ~ 10% CPU, bandwidth ~ 300MB/s
Kernel 5.14rc6 source local, target remote => ~ 2.2% CPU, bandwidth ~ 680MB/s
Kernel 5.11 source remote, target local => ~ 7% CPU, bandwidth ~ 870MB/s
Kernel 5.11 source local, target remote => ~ 2.7% CPU, bandwidth ~ 1050MB/s
Where 100% CPU means 256 cores used. On 5.11 it looks like the application could scale to about 10-12GB/s if data would be copied from remote or even reach full network bandwidth (25GB/s) if data is pushed to remote. There is room for improvement I'd say for remote to local, but 10GB/s is already impressive enough. Sorry for wasting time. Since kernel 5.14 fixes a huge mdadm performance bottleneck, would have thought it cannot be bad for networking.
That's a great find. Thanks for sharing as well.
We've noticed networking problems before with newer kernels so I tend to always just stay back.
Cannot edit the previous post, but realized it could be important for whoever may look into throughput numbers or similar issues: metrics were obtained with "--checkers=32 --transfers 2 --multi-thread-streams=0" and the tool looks to scale well with transfers or multi-threaded-streams increases as long as you can read/write fast enough.
These should be nice low CPU usage, so we'll have to look elsewhere for the problem...
Wow, that is quite variable and quite poor on rc6.
Hmm, I wonder if rc6 messed up the futex implementation somehow.
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.