Thanks for the suggestion on disabling checksum! I'll give that a try.
Since rclone serve http is fast enough, I will like use that over SFTP, but it seems odd that there is such a large difference in speed. I would like SFTP as an option for my clients, though, in case that is the preferred interface. Access to the S3 bucket is not an issue, as I will make it read-only - this scenario is purely for copying data out.