File Corruption with Mixed Content Issue during SFTP to AWS S3 Transfer using rclone copy

Help and Support
1

What is the problem you are having with rclone?

When employing the rclone copy command to transfer files from an SFTP server to AWS S3, an issue was encountered where one of the files appeared to be corrupted. Although the size of the corrupted file matched the original file, its content included data from other files.

Run the command 'rclone version' and share the full output of the command.

rclone v1.65.2

  • os/version: ubuntu 22.04 (64 bit)
  • os/kernel: 6.1.66 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.21.6
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

AWS S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy sftp:/root s3:bucket1/vendor --max-age 3d

The rclone config contents with secrets removed.

{
    "sftp": {
        "host": "files.xxxxx.com",
        "md5sum_command": "none",
        "sha1sum_command": "none",
        "shell_type": "unix",
        "type": "sftp",
        "user": "user1"
    },
    "s3": {
        "acl": "private",
        "env_auth": "true",
        "location_constraint": "us-east-2",
        "provider": "AWS",
        "region": "us-east-2",
        "storage_class": "STANDARD",
        "type": "s3"
    }
}

A log from the command with the -vv flag

log was missing
2

Try to replicate the issues and capture debug log:

rclone copy sftp:/root s3:bucket1/vendor --max-age 3d -vv --log-file /path/to/rclone.log

Then provide some evidence which files are different and you think should be the same. You could download them and generate hashes for example.

3

This is an occasional problem and I have yet to have it happen again.

After the problem occurred, I manually deleted the target file and then used rclone to resynchronize the data, and the result was correct. The main error is that the file sizes are exactly the same.

4

So until you have some facts it is only a story. It is impossible to investigate anything.

If true it would be very serious bug indeed.

I suggest you enable MD5 hash on your sftp source. MD5 is also supported by S3 and would allow rclone to validate transferred files automatically.

5

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.