How does rclone verify file transfer - s3 to s3

Help and Support
1

What is the problem you are having with rclone?

no problem, asking question.

took a look at the source code and i wanted to confirm

for a multi-part upload file copy, from s3 bucket to s3 bucket.
how does rclone ensure that the dest file matches the source?

does rclone compare the dest ETag to the source ETag?
or is rclone performing a running calculation and using that?

Run the command 'rclone version' and share the full output of the command.

rclone v1.57.0
- os/version: Microsoft Windows 10 Pro 2009 (64 bit)
- os/kernel: 10.0.22000.493 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.17.2
- go/linking: dynamic
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

for testing, both remotes are wasabi s3, each remote is using a unique bucket in a different endpoint.
for production, a large amount of TB between wasabi to aws s3 deep glacier.

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy source:zorksource/file.txt dest:zorkdest --s3-chunk-size=5Mi --s3-upload-concurrency=1 --s3-upload-cutoff=1b --s3-copy-cutoff=1b --s3-no-check-bucket --dump=headers --retries=1 --low-level-retries=1 --stats=0 --log-level=DEBUG --log-file=.\\log.source2dest.txt

The rclone config contents with secrets removed.

[source]
type = s3
provider = Wasabi
access_key_id = 
secret_access_key = 
endpoint = s3.us-east-2.wasabisys.com

[dest]
type = s3
provider = Wasabi
access_key_id = 
secret_access_key = 
endpoint = s3.us-west-1.wasabisys.com

A log from the command with the -vv flag

DEBUG : rclone: Version "v1.57.0" starting with parameters ["c:\\data\\rclone\\scripts\\rclone.exe" "copy" "source:zorksource/file.txt" "dest:zorkdest" "--s3-chunk-size=5Mi" "--s3-upload-concurrency=1" "--s3-upload-cutoff=1b" "--s3-copy-cutoff=1b" "--s3-no-check-bucket" "--dump=headers" "--retries=1" "--low-level-retries=1" "--stats=0" "--log-level=DEBUG" "--log-file=.\\\\log.source2dest.txt"]
DEBUG : Creating backend with remote "source:zorksource/file.txt"
DEBUG : Using config file from "c:\\data\\rclone\\scripts\\barantec.conf"
DEBUG : source: detected overridden config - adding "{8c8ZL}" suffix to name
DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : HTTP REQUEST (req 0xc0002aa700)
DEBUG : HEAD /zorksource/file.txt HTTP/1.1
Host: s3.us-east-2.wasabisys.com
User-Agent: rclone/v1.57.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220213T234341Z

DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : HTTP RESPONSE (req 0xc0002aa700)
DEBUG : HTTP/1.1 200 OK
Content-Length: 6291456
Accept-Ranges: bytes
Content-Type: text/plain; charset=utf-8
Date: Sun, 13 Feb 2022 23:43:42 GMT
Etag: "41b392acc7c51164aee62c3b96386db0-2"
Last-Modified: Sun, 13 Feb 2022 23:43:42 GMT
Server: WasabiS3/7.2.3941-2022-02-03-4a1b388 (head5)
X-Amz-Id-2: YVF9//Tyzt8BImh7VHD8At3+UXbNJKuQh3ewq9qvMkK3X2wjLxJD0GBLlrVzbG9x9xjgfGu9P9Lp
X-Amz-Meta-Md5chksum: /Ic8BG9v3xOE0suoBuadbA==
X-Amz-Meta-Mtime: 1644783735.4892583
X-Amz-Request-Id: D98F2FDB790D8114

DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : fs cache: adding new entry for parent of "source:zorksource/file.txt", "source{8c8ZL}:zorksource"
DEBUG : Creating backend with remote "dest:zorkdest"
DEBUG : dest: detected overridden config - adding "{8c8ZL}" suffix to name
DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
DEBUG : fs cache: renaming cache item "dest:zorkdest" to be canonical "dest{8c8ZL}:zorkdest"
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : HTTP REQUEST (req 0xc00072aa00)
DEBUG : HEAD /zorksource/file.txt HTTP/1.1
Host: s3.us-east-2.wasabisys.com
User-Agent: rclone/v1.57.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220213T234341Z

DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : HTTP RESPONSE (req 0xc00072aa00)
DEBUG : HTTP/1.1 200 OK
Content-Length: 6291456
Accept-Ranges: bytes
Content-Type: text/plain; charset=utf-8
Date: Sun, 13 Feb 2022 23:43:42 GMT
Etag: "41b392acc7c51164aee62c3b96386db0-2"
Last-Modified: Sun, 13 Feb 2022 23:43:42 GMT
Server: WasabiS3/7.2.3941-2022-02-03-4a1b388 (head5)
X-Amz-Id-2: rtNfvt0/C6UBIlNG+RHK/BMZVXkuS2U6uPLtd1zXaSi97kNLXyGur1E57DUzQTg/wxGmFn2APBex
X-Amz-Meta-Md5chksum: /Ic8BG9v3xOE0suoBuadbA==
X-Amz-Meta-Mtime: 1644783735.4892583
X-Amz-Request-Id: DE69F657A35FF116

DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : HTTP REQUEST (req 0xc0008b2500)
DEBUG : HEAD /zorkdest/file.txt HTTP/1.1
Host: s3.us-west-1.wasabisys.com
User-Agent: rclone/v1.57.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220213T234341Z

DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : HTTP RESPONSE (req 0xc0008b2500)
DEBUG : HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Sun, 13 Feb 2022 23:43:43 GMT
Server: WasabiS3/7.2.3941-2022-02-03-4a1b388 (head10)
X-Amz-Id-2: dqKwzefEW5cnvAaZDB+MG95LlQEo67fc32/Fx6wx8mKsY7e9kiya8B9hydj11Cuv9s492ll0cO8r
X-Amz-Request-Id: 46522B47289FFAFC

DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : file.txt: Need to transfer - File not found at Destination
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : HTTP REQUEST (req 0xc0002aac00)
DEBUG : GET /zorksource/file.txt HTTP/1.1
Host: s3.us-east-2.wasabisys.com
User-Agent: rclone/v1.57.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220213T234341Z
Accept-Encoding: gzip

DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : HTTP RESPONSE (req 0xc0002aac00)
DEBUG : HTTP/1.1 200 OK
Content-Length: 6291456
Accept-Ranges: bytes
Content-Type: text/plain; charset=utf-8
Date: Sun, 13 Feb 2022 23:43:43 GMT
Etag: "41b392acc7c51164aee62c3b96386db0-2"
Last-Modified: Sun, 13 Feb 2022 23:43:42 GMT
Server: WasabiS3/7.2.3941-2022-02-03-4a1b388 (head5)
X-Amz-Id-2: 7BuD7zs5o559Z4McBtJWk2dWRt0Nk0vzPQBEkbb+mYhbF2pLb3fvS8hkNICUwC1v3SvSLrrEKCl3
X-Amz-Meta-Md5chksum: /Ic8BG9v3xOE0suoBuadbA==
X-Amz-Meta-Mtime: 1644783735.4892583
X-Amz-Request-Id: 9238FEBDB60FCACF

DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : HTTP REQUEST (req 0xc000198000)
DEBUG : POST /zorkdest/file.txt?uploads= HTTP/1.1
Host: s3.us-west-1.wasabisys.com
User-Agent: rclone/v1.57.0
Content-Length: 0
Authorization: XXXX
Content-Type: text/plain; charset=utf-8
X-Amz-Acl: private
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220213T234342Z
X-Amz-Meta-Md5chksum: /Ic8BG9v3xOE0suoBuadbA==
X-Amz-Meta-Mtime: 1644783735.4892583
Accept-Encoding: gzip

DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : HTTP RESPONSE (req 0xc000198000)
DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Date: Sun, 13 Feb 2022 23:43:43 GMT
Server: WasabiS3/7.2.3941-2022-02-03-4a1b388 (head10)
X-Amz-Id-2: U571W0ZJ0WrPlRRrqAC9QQQlZiQfYoAOl3L/a49Xh3LQ+LN8T3qQwCBxZzkyDHClqNisbHo1mqG7
X-Amz-Request-Id: EB5FE36FCBE8F9E0

DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : file.txt: multipart upload starting chunk 1 size 5Mi offset 0/6Mi
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : HTTP REQUEST (req 0xc00072a100)
DEBUG : PUT /zorkdest/file.txt?partNumber=1&uploadId=AfpDu1N9bmy3tAgoW2YlZHT4bPZgdsgVTPKAsYoalWcufVqKVgNvfOED7sTiEF9VukkAwraXnEA13yoIZAOVc48n7s0agsbqpUBL7KICe7hUIY9iHN3WVD212SiJBVQ1 HTTP/1.1
Host: s3.us-west-1.wasabisys.com
User-Agent: rclone/v1.57.0
Content-Length: 5242880
Authorization: XXXX
Content-Md5: YwohxPIQ30r59ZKUjSA5hQ==
Expect: 100-Continue
X-Amz-Content-Sha256: 0a48bce725f726914568e9497b4b8417585773ce0330cee548c2f4b4f3b14867
X-Amz-Date: 20220213T234342Z
Accept-Encoding: gzip

DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : HTTP RESPONSE (req 0xc00072a100)
DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Sun, 13 Feb 2022 23:43:45 GMT
Etag: "630a21c4f210df4af9f592948d203985"
Server: WasabiS3/7.2.3941-2022-02-03-4a1b388 (head10)
X-Amz-Id-2: f6l66O5sM5+uqScb5W6TvSIHSOKyy0zyD24madIf68cFiZe4laIhbjLQ0zPUtr+5g4swGWOVV5LT
X-Amz-Request-Id: 74D1C31BC1203A59

DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : file.txt: multipart upload starting chunk 2 size 1Mi offset 5Mi/6Mi
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : HTTP REQUEST (req 0xc00072a700)
DEBUG : PUT /zorkdest/file.txt?partNumber=2&uploadId=AfpDu1N9bmy3tAgoW2YlZHT4bPZgdsgVTPKAsYoalWcufVqKVgNvfOED7sTiEF9VukkAwraXnEA13yoIZAOVc48n7s0agsbqpUBL7KICe7hUIY9iHN3WVD212SiJBVQ1 HTTP/1.1
Host: s3.us-west-1.wasabisys.com
User-Agent: rclone/v1.57.0
Content-Length: 1048576
Authorization: XXXX
Content-Md5: 0eVmJbdTd7pOjMMMSFQ14g==
X-Amz-Content-Sha256: ef7eae4416a1bd60b9f1c65e0cf68dc3d6d5bb9c21967883402d67f9c81608c7
X-Amz-Date: 20220213T234344Z
Accept-Encoding: gzip

DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : HTTP RESPONSE (req 0xc00072a700)
DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Sun, 13 Feb 2022 23:43:46 GMT
Etag: "d1e56625b75377ba4e8cc30c485435e2"
Server: WasabiS3/7.2.3941-2022-02-03-4a1b388 (head10)
X-Amz-Id-2: Mr3gbZ7OHCCGBMVRP9QMpmnFFW8koTQoDADuF8XYQWGjHpWXW48bAKRP/ZhWf40OMGh8e6dW8SuV
X-Amz-Request-Id: 5C30696FDB80155C

DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : HTTP REQUEST (req 0xc0000cba00)
DEBUG : POST /zorkdest/file.txt?uploadId=AfpDu1N9bmy3tAgoW2YlZHT4bPZgdsgVTPKAsYoalWcufVqKVgNvfOED7sTiEF9VukkAwraXnEA13yoIZAOVc48n7s0agsbqpUBL7KICe7hUIY9iHN3WVD212SiJBVQ1 HTTP/1.1
Host: s3.us-west-1.wasabisys.com
User-Agent: rclone/v1.57.0
Content-Length: 287
Authorization: XXXX
X-Amz-Content-Sha256: 80b32595fec8ff5be7c3c8507e10db819d70c6bc538354e2bc9ece3db8de5fc7
X-Amz-Date: 20220213T234344Z
Accept-Encoding: gzip

DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : HTTP RESPONSE (req 0xc0000cba00)
DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Sun, 13 Feb 2022 23:43:46 GMT
Server: WasabiS3/7.2.3941-2022-02-03-4a1b388 (head10)
X-Amz-Id-2: pXMXV0lSRvvFqbhHNDvZHvUS0LsUt/MxD/bHjMk0uco2z9yDHNzeQ3lfAtOBMtamtbmqz/P+sy9C
X-Amz-Request-Id: D53D32A2969BC8AB

DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : HTTP REQUEST (req 0xc0008b2400)
DEBUG : HEAD /zorkdest/file.txt HTTP/1.1
Host: s3.us-west-1.wasabisys.com
User-Agent: rclone/v1.57.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220213T234344Z

DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : HTTP RESPONSE (req 0xc0008b2400)
DEBUG : HTTP/1.1 200 OK
Content-Length: 6291456
Accept-Ranges: bytes
Content-Type: text/plain; charset=utf-8
Date: Sun, 13 Feb 2022 23:43:46 GMT
Etag: "41b392acc7c51164aee62c3b96386db0-2"
Last-Modified: Sun, 13 Feb 2022 23:43:46 GMT
Server: WasabiS3/7.2.3941-2022-02-03-4a1b388 (head10)
X-Amz-Id-2: W2AH9A8T2I3ptS5TpnbeEJ9bvY3A5zFxLEpkncfFKzOYk9ibyEJsg+iz5Svibc+9ZZ5ojVIUfrPj
X-Amz-Meta-Md5chksum: /Ic8BG9v3xOE0suoBuadbA==
X-Amz-Meta-Mtime: 1644783735.4892583
X-Amz-Request-Id: 6438DD0BEF1A64DC

DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : file.txt: md5 = fc873c046f6fdf1384d2cba806e69d6c OK
INFO  : file.txt: Copied (new)
DEBUG : 10 go routines active
2

For a multipart upload the etag is not an md5.

However, if the md5 is known at the time of upload (which it will be for the local disk) then rclone adds some metadata with it in to the object. You can see that here

Rclone can then use that for checks.

It is not as good a check as the etag, but it is still useful at detecting various kinds of bit rot.

3

thanks
so rclone copy, s3 -> s3, has no way to ensure the dest matches the source.

rclone relies
--- on the https protocol
--- Content-Md5: for each chunk transferred?

so with s3 -> s3, this output cannot be used to verify the file transfer?
DEBUG : file.txt: md5 = fc873c046f6fdf1384d2cba806e69d6c OK

4

Not during the copy. However each chunk it protected with a sha256 if I remember rightly.

Yes

that is correct. I can't remember if rclone sets Content-MD5 for multipart but it definitely sets it for single part

That's the md5 of the source but rclone hasn't had a confirmation from s3 that it is correct for multipart uploads.

It occurs to me that we could calculate what the etag should be during the upload and check that. That would be the bullet proof check you were hoping for. For multipart uploads it isn't an md5 but an MD5 of md5s which we can calculate as we upload as we know the chunk size.

5

yes, bullet proof.

based on --dump=bodies, seems that the answer is yes.
tho i do not know what is calculating Content-MD5, rclone or the underlying aws s3 sdk.

i assume that local to s3 has the same lack of bulletproof.
perhaps in this case
--- calculate md5. calculate ETag based on chunk size
--- store md5 as X-Amz-Meta-Md5chksum and upload file
--- compare calculated Etag to the backend provider ETag.

6

I think it's worth having a go with this idea.

Do you fancy making an issue about it?

7

here it is
use ETag to verify S3 file transfers · Issue #5993 · rclone/rclone · GitHub

i think this is a major upgrade for rclone's support for s3.

looking forward to beta testing the blank out of it.

8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.