How does rclone verify file transfer - s3 to s3

[ncw (Nick Craig-Wood)]

@asdffdsa

so rclone copy, s3 -> s3, has no way to ensure the dest matches the source.

Not during the copy. However each chunk it protected with a sha256 if I remember rightly.

@asdffdsa

rclone relies
--- on the https protocol
--- Content-Md5: for each chunk transferred?

Yes

that is correct. I can't remember if rclone sets Content-MD5 for multipart but it definitely sets it for single part

@asdffdsa

with s3 -> s3, this output cannot be used to verify the file transfer?
DEBUG : file.txt: md5 = fc873c046f6fdf1384d2cba806e69d6c OK

That's the md5 of the source but rclone hasn't had a confirmation from s3 that it is correct for multipart uploads.

It occurs to me that we could calculate what the etag should be during the upload and check that. That would be the bullet proof check you were hoping for. For multipart uploads it isn't an md5 but an MD5 of md5s which we can calculate as we upload as we know the chunk size.