I have managed to discover what is going on.
In sftp: Update ReadFrom failed: failed to send packet: EOF errors and very slow · Issue #5197 · rclone/rclone · GitHub I have posted a new test binary which should fix the problem - testing appreciated!
It it works I'll put it in 1.55.1