SFTP and set_modtime=false Error

What is the problem you are having with rclone?

When I run rclone to copy files to an SFTP remote, and disable the post-transfer file modification time update, I get a "corrupted on transfer" error. A work-around is to use the --ignore-size flag. Just wondering if this is by design, or if rclone should be doing a post-transfer "stat" on the remote file to capture size at least?

What is your rclone version (output from rclone version)

rclone v1.51.0

  • os/arch: linux/amd64
  • go version: go1.13.7

Which OS you are using and how many bits (eg Windows 7, 64 bit)

CentOS 7, 64-bit

Which cloud storage system are you using? (eg Google Drive)

SFTP

The command you were trying to run (eg rclone copy /tmp remote:tmp)

The following command copies from local to SFTP, where SFTP is the same machine and user.

rclone copy ./test :sftp:work/foobar -P --sftp-set-modtime=false --sftp-key-file <ssh key file> --retries 1

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

2020-05-13 15:49:24 ERROR : file1.txt: corrupted on transfer: sizes differ 6 vs 0
2020-05-13 15:49:25 ERROR : file2.txt: corrupted on transfer: sizes differ 6 vs 0
2020-05-13 15:49:25 ERROR : Attempt 1/1 failed with 2 errors and: corrupted on transfer: sizes differ 6 vs 0
Transferred:            12 / 12 Bytes, 100%, 41 Bytes/s, ETA 0s
Errors:                 2 (retrying may help)
Elapsed time:         0.2s

Hmm! so it does!

Yes this looks like a bug.

Try this

https://beta.rclone.org/branch/v1.51.0-321-g568c4d3b-fix-sftp-size-beta/ (uploaded in 15-30 mins)

@ncw Sorry...I can't find the beta build. Can you re-post?

Sorry, forgot to push it!

https://beta.rclone.org/branch/v1.51.0-322-gc3c225cc-fix-sftp-size-beta/ (uploaded in 15-30 mins)

@ncw Works great now! In my use-case, we are actually trying to transfer files to a SFTP destination that is backed by s3fs-fuse. One consequence of setting the file mod time after transfer is that s3fs-fuse will make a CopyObject call to AWS S3 to set the object metadata. For large files, this was causing too much of a delay to make having an accurate file mod time worth it. Thanks for the fix!

Will the next version of rclone be released soon? There's some lead time for me to get approval to use it.

Great!

I've merged this to master now which means it will be in the latest beta in 15-30 mins and released in v1.52

Interesting!

You know that rclone can transfer direct to s3 I'm sure... rclone copy /path s3:bucket

Also rclone can serve sftp directly from s3 also rclone serve sftp s3:bucket

Make sense - rclone would do the same!

Should be this weekend :crossed_fingers:

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.