The point is that checksum also runs on source where operation involves reading from disk at 1GB/s, doing md5, then pushing the data over network. The source has now an average usage of about of ~3% usage (~7.5 cores) for last hours.
Right now target has 23,4% usage (~60 cores) hogged by rclone. Disk array is based on NVMe, read await and write await are under 1ms under peaks. It is definitely a CPU problem, not a disk one, as I can read at 70GB/s for less than 10% machine load. Writing is the same. If scaling linearly with source load, I should get 10GB/s at 30% but bottleneck appears to be on target on rclone side. Are there any known performance metrics of how much is expected for 100MB/s traffic?