Rclone serve sftp + WinSCP background mode uploading causes file corruption

What is the problem you are having with rclone?

When using rclone serve sftp <local directory> with WinSCP as the client, uploaded file in background mode is corrupted (MD5 hash mismatch; file part at the beginning reset to zero-bits).

Steps to reproduce:

  • Run rclone sftp server using the command line below.
  • Open WinSCP and connect to it.
  • Drag-and-drop a relatively large file (mine is a 19.9 MiB file) into the remote directory view. In the halfway of the upload progress, click "Process In Background" button.
  • Wait for it to finish uploading.
  • Examine the uploaded file at the server side. It differs from the original file, and a large part of it at the beginning of the file is reset to all zero bits. The percentage of the part approximates the progress where the upload process is scheduled to background.

If the upload is not scheduled to background, the upload file seems intact.

I don't know if it is an issue of WinSCP or rclone, but WinSCP works well with OpenSSH SFTP server with no such problem.

Run the command 'rclone version' and share the full output of the command.

rclone v1.75.0-beta.9756.7dcbc8b4f

  • os/version: debian 12.14 (64 bit)
  • os/kernel: 6.1.0-48-amd64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.26.3
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

SFTP serve mode serving a local directory

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone serve sftp -v -v --user user --authorized-keys /path/to/authorized_keys --addr :30022 --copy-links --vfs-cache-mode=writes --dir-cache-time 0m3s --poll-interval 0m3s --local-encoding=Slash,InvalidUtf8 /srv/test-rclone

The rclone config contents with secrets removed.

No config file. /home/user/.config/rclone/rclone.conf does not exist.
No RCLONE_* environment variable exists.
Everything is provided through command line.

A log from the command with the -vv flag

https://pastebin.com/6uUy4PKG

Thanks for reporting this. Yes this is a bug in rclone to do with it ignoring the Open flags the sftp client is sending which means the file is being truncated when the background operation starts. This explains the corruption (the zero bytes at the start) perfectly.

Here is a fix where we obey the open flags properly - please give it a go!

v1.75.0-beta.9757.9c9d78ce3.fix-sftp-serve-winscp-resume on branch fix-sftp-serve-winscp-resume (uploaded in 15-30 mins)

I did a review of the serve sftp source for unimplemented features and I found a few

  • open flags (which caused your corruption)
  • size ignored in truncate
  • statvfs (shows disk usage - it is implemented in the VFS - I don't know why we never plumbed it in - maybe it didn't exist when we made this command in 2019)

So can you try this one instead? Thanks

v1.75.0-beta.9760.bcb1401c1.fix-sftp-serve-winscp-resume on branch fix-sftp-serve-winscp-resume (uploaded in 15-30 mins)

I can confirm v1.75.0-beta.9760.bcb1401c1.fix-sftp-serve-winscp-resume fixes the problem. Thanks!

Thanks for testing @shunf4

I've merged this to master now which means it will be in the latest beta in 15-30 minutes and released in v1.75

If we have another point release I'll put the corruption fix in v1.74.3