In some cases, rclone does not use ETag to verify files

asdffdsa · March 8, 2023, 2:46pm

thanks, sorry for the delay,

i was waiting to hear back from wasabi,

Thank you for the report on the issue - it is appreciated! I'll be sure to keep you posted on any updates for this on our end, 
although this would likely be a low-priority bug to work on

i did the testing with AWS

seems that the rclone did not calculate the correct ETag.
Etag differ: expecting 5fa6d124612290af0c85759cb50ce07a-10 but got 0b844664e3e49247b7ed8ae0871ae399-10

on the other hand, my bash script, as before, was able to calculate the ETag.

note: the trick is to use --s3-upload-concurrency=1, then the individual Etags arrive in correct sequence.
when using --s3-upload-concurrency greater than 1, the individual Etags can arrive out of sequence.

etag parts
-------------------------------
cb7accdee5d3207e9411f25c7956f092  -
de3418dbe826686b6962ff91e00dfc0a  -
a22fa1c8c88c232d3d019802a8bcfb61  -
80f92f603c76b3e149d9c01a83f3c5a0  -
c24c827c73e69d2432111f75740d59d3  -
b8e84128e190eec22ce3c1308bc158ee  -
220bb561aa9730f16f4b8074bd9eb549  -
e615b5263a49cde19c6f9868fb5c3441  -
46e790723154d9b45821f6a1b6b7c9c2  -
b8d1cc8eda9c0104d4575c6dd774aec6  -
-------------------------------

etag_from_aws   = 0b844664e3e49247b7ed8ae0871ae399-10
etag_calculated = 0b844664e3e49247b7ed8ae0871ae399-10
The ETags match

and here is the debug log

2023/03/08 09:36:51 DEBUG : rclone: Version "v1.62.0-beta.6741.f793faae4.fix-s3-sse-multipart" starting with parameters ["./rclone" "copy" "./source" "aws_ssec.and.etag:" "--s3-upload-concurrency=1" "--s3-no-head" "--s3-upload-cutoff=0" "--s3-chunk-size=10M" "-vv" "--retries=1" "--files-from=missing-on-dst.txt" "--dump=headers,responses" "--log-file=debug_aws_ssec.txt"]
2023/03/08 09:36:51 DEBUG : Creating backend with remote "./source"
2023/03/08 09:36:51 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2023/03/08 09:36:51 DEBUG : fs cache: renaming cache item "./source" to be canonical "/root/rclone/scripts/minimal/source"
2023/03/08 09:36:51 DEBUG : Creating backend with remote "aws_ssec.and.etag:"
2023/03/08 09:36:51 DEBUG : Creating backend with remote "aws_ssec.and.etag_remote:ssec.and.etag"
2023/03/08 09:36:51 DEBUG : aws_ssec.and.etag_remote: detected overridden config - adding "{bsCrg}" suffix to name
2023/03/08 09:36:51 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2023/03/08 09:36:51 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2023/03/08 09:36:51 DEBUG : fs cache: renaming cache item "aws_ssec.and.etag_remote:ssec.and.etag" to be canonical "aws_ssec.and.etag_remote{bsCrg}:ssec.and.etag"
2023/03/08 09:36:51 DEBUG : fs cache: renaming cache item "aws_ssec.and.etag:" to be canonical "aws_ssec.and.etag_remote{bsCrg}:ssec.and.etag"
2023/03/08 09:36:51 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:51 DEBUG : HTTP REQUEST (req 0xc000a7a800)
2023/03/08 09:36:51 DEBUG : GET /ssec.and.etag?delimiter=%2F&encoding-type=url&list-type=2&max-keys=1000&prefix= HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6741.f793faae4.fix-s3-sse-multipart
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20230308T143651Z
Accept-Encoding: gzip

2023/03/08 09:36:51 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:51 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:36:51 DEBUG : HTTP RESPONSE (req 0xc000a7a800)
2023/03/08 09:36:51 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Wed, 08 Mar 2023 14:36:53 GMT
Server: AmazonS3
X-Amz-Bucket-Region: us-east-1
X-Amz-Id-2: 4k//8l02ShmhKopSNNiVYU71u2EWzDpzo8ujKbtshqmVFH9RCalTy2PtFn7DmFD9lQf6vqm0KdQ=
X-Amz-Request-Id: ST09NFTJC51M5BQR

12c
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>ssec.and.etag</Name><Prefix></Prefix><KeyCount>0</KeyCount><MaxKeys>1000</MaxKeys><Delimiter>/</Delimiter><EncodingType>url</EncodingType><IsTruncated>false</IsTruncated></ListBucketResult>
0

2023/03/08 09:36:51 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:36:51 DEBUG : S3 bucket ssec.and.etag: Waiting for checks to finish
2023/03/08 09:36:51 DEBUG : S3 bucket ssec.and.etag: Waiting for transfers to finish
2023/03/08 09:36:52 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:52 DEBUG : HTTP REQUEST (req 0xc00084c500)
2023/03/08 09:36:52 DEBUG : POST /ssec.and.etag/file.ext?uploads= HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6741.f793faae4.fix-s3-sse-multipart
Content-Length: 0
Authorization: XXXX
Content-Type: application/octet-stream
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20230308T143652Z
X-Amz-Meta-Md5chksum: 4NV7wWg7w2fq8IHaowJf3w==
X-Amz-Meta-Mtime: 1678286111.4957453
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key: MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
X-Amz-Storage-Class: STANDARD
Accept-Encoding: gzip

2023/03/08 09:36:52 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:52 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:36:52 DEBUG : HTTP RESPONSE (req 0xc00084c500)
2023/03/08 09:36:52 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Date: Wed, 08 Mar 2023 14:36:54 GMT
Server: AmazonS3
X-Amz-Id-2: q1VU+k1iI7kDjaYXedqeh4wK6s6f8hR8eYVNKxaRxzyGJLYInpK7xrkuZlhHG8AT9nVvAcB+H0Q=
X-Amz-Request-Id: H6CHC8AAY18V76S2
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==

134
<?xml version="1.0" encoding="UTF-8"?>
<InitiateMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Bucket>ssec.and.etag</Bucket><Key>file.ext</Key><UploadId>FDcCY2scQU_RorCofQ4Hyoh3xND0B7ad387Sao49.BDyPQfK5syqUMPpam.nR.BlfnOjmvs.kq4kOV8BpwMYwg--</UploadId></InitiateMultipartUploadResult>
0

2023/03/08 09:36:52 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:36:52 DEBUG : file.ext: multipart upload starting chunk 1 size 10Mi offset 0/100Mi
2023/03/08 09:36:52 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:52 DEBUG : HTTP REQUEST (req 0xc00084c900)
2023/03/08 09:36:52 DEBUG : PUT /ssec.and.etag/file.ext?partNumber=1&uploadId=FDcCY2scQU_RorCofQ4Hyoh3xND0B7ad387Sao49.BDyPQfK5syqUMPpam.nR.BlfnOjmvs.kq4kOV8BpwMYwg-- HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6741.f793faae4.fix-s3-sse-multipart
Content-Length: 10485760
Authorization: XXXX
Content-Md5: Z3h/lNXi14ku6xzT5Wq2Hw==
Expect: 100-Continue
X-Amz-Content-Sha256: 34fa91a511f950ea0c2a4e0b0fe483f4f472b1ac2314ef419148719956d9d487
X-Amz-Date: 20230308T143652Z
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key: MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
Accept-Encoding: gzip

2023/03/08 09:36:52 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:53 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:36:53 DEBUG : HTTP RESPONSE (req 0xc00084c900)
2023/03/08 09:36:53 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Wed, 08 Mar 2023 14:36:54 GMT
Etag: "cb7accdee5d3207e9411f25c7956f092"
Server: AmazonS3
X-Amz-Id-2: u2iN0+KEe7gREqwpwhFX8jZBDkEKwUnenBuLPnLnQ2Q96F18ENfDI7RAsZCLp/bCuafH9GswBro=
X-Amz-Request-Id: H6CGVS611J4NNZRT
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==

2023/03/08 09:36:53 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:36:53 DEBUG : file.ext: multipart upload starting chunk 2 size 10Mi offset 10Mi/100Mi
2023/03/08 09:36:53 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:53 DEBUG : HTTP REQUEST (req 0xc00084c400)
2023/03/08 09:36:53 DEBUG : PUT /ssec.and.etag/file.ext?partNumber=2&uploadId=FDcCY2scQU_RorCofQ4Hyoh3xND0B7ad387Sao49.BDyPQfK5syqUMPpam.nR.BlfnOjmvs.kq4kOV8BpwMYwg-- HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6741.f793faae4.fix-s3-sse-multipart
Content-Length: 10485760
Authorization: XXXX
Content-Md5: +BBIKFxd1xMMw09diQAXug==
Expect: 100-Continue
X-Amz-Content-Sha256: e181a7bf70f327ebeda3886021879d2c56822db69845c4bbdf267e1a410196ad
X-Amz-Date: 20230308T143653Z
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key: MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
Accept-Encoding: gzip

2023/03/08 09:36:53 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:53 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:36:53 DEBUG : HTTP RESPONSE (req 0xc00084c400)
2023/03/08 09:36:53 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Wed, 08 Mar 2023 14:36:55 GMT
Etag: "de3418dbe826686b6962ff91e00dfc0a"
Server: AmazonS3
X-Amz-Id-2: jPHj0+tgwz/PnyxPN+RLwvP8JqTFYUXutyyCXmx6lFr7BrCOvwO4+1XTfBvCzVLtSt5rJP6ukn0=
X-Amz-Request-Id: HTBF7FMCRKPMBVZN
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==

2023/03/08 09:36:53 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:36:53 DEBUG : file.ext: multipart upload starting chunk 3 size 10Mi offset 20Mi/100Mi
2023/03/08 09:36:53 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:53 DEBUG : HTTP REQUEST (req 0xc000a7a100)
2023/03/08 09:36:53 DEBUG : PUT /ssec.and.etag/file.ext?partNumber=3&uploadId=FDcCY2scQU_RorCofQ4Hyoh3xND0B7ad387Sao49.BDyPQfK5syqUMPpam.nR.BlfnOjmvs.kq4kOV8BpwMYwg-- HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6741.f793faae4.fix-s3-sse-multipart
Content-Length: 10485760
Authorization: XXXX
Content-Md5: jSDg6nOjQCqlg004CtnszA==
Expect: 100-Continue
X-Amz-Content-Sha256: c7b401be36ba98f434fdfda220a719a561bc762c122fa93f34b04867c7d45cb0
X-Amz-Date: 20230308T143653Z
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key: MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
Accept-Encoding: gzip

2023/03/08 09:36:53 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:54 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:36:54 DEBUG : HTTP RESPONSE (req 0xc000a7a100)
2023/03/08 09:36:54 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Wed, 08 Mar 2023 14:36:55 GMT
Etag: "a22fa1c8c88c232d3d019802a8bcfb61"
Server: AmazonS3
X-Amz-Id-2: hUIYZJ90vRfnjaTswMvpghuiXkxClDxKNWZJTDOlC9o3kNl08IYft+zkX7xSADZGj1fmi9C41ao=
X-Amz-Request-Id: HTB5BJ9EHQW5NY63
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==

2023/03/08 09:36:54 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:36:54 DEBUG : file.ext: multipart upload starting chunk 4 size 10Mi offset 30Mi/100Mi
2023/03/08 09:36:54 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:54 DEBUG : HTTP REQUEST (req 0xc00084c400)
2023/03/08 09:36:54 DEBUG : PUT /ssec.and.etag/file.ext?partNumber=4&uploadId=FDcCY2scQU_RorCofQ4Hyoh3xND0B7ad387Sao49.BDyPQfK5syqUMPpam.nR.BlfnOjmvs.kq4kOV8BpwMYwg-- HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6741.f793faae4.fix-s3-sse-multipart
Content-Length: 10485760
Authorization: XXXX
Content-Md5: 8BQVb4Z9X/xItJ7mM9RSog==
Expect: 100-Continue
X-Amz-Content-Sha256: d1b701e89dd402e58e10b2b0572121a1f01e0da304725b3662bdf312dc529add
X-Amz-Date: 20230308T143654Z
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key: MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
Accept-Encoding: gzip

2023/03/08 09:36:54 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:55 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:36:55 DEBUG : HTTP RESPONSE (req 0xc00084c400)
2023/03/08 09:36:55 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Wed, 08 Mar 2023 14:36:56 GMT
Etag: "80f92f603c76b3e149d9c01a83f3c5a0"
Server: AmazonS3
X-Amz-Id-2: ckWh3jJ8DgeAJPYXyj6RT3bvXGlZcZh8Kw7vsqehdpNZJr6JWfo8eAj3QpQ0SU7flZ72sn1NNEE=
X-Amz-Request-Id: M1TE9KD660ZDZ82M
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==

2023/03/08 09:36:55 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:36:55 DEBUG : file.ext: multipart upload starting chunk 5 size 10Mi offset 40Mi/100Mi
2023/03/08 09:36:55 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:55 DEBUG : HTTP REQUEST (req 0xc00084c800)
2023/03/08 09:36:55 DEBUG : PUT /ssec.and.etag/file.ext?partNumber=5&uploadId=FDcCY2scQU_RorCofQ4Hyoh3xND0B7ad387Sao49.BDyPQfK5syqUMPpam.nR.BlfnOjmvs.kq4kOV8BpwMYwg-- HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6741.f793faae4.fix-s3-sse-multipart
Content-Length: 10485760
Authorization: XXXX
Content-Md5: +6WJre7P2U5ftVd0b0qQrg==
Expect: 100-Continue
X-Amz-Content-Sha256: e99abe78681edcd89aea2e0e967152ffe4609e06b3528d751cc814daf874932e
X-Amz-Date: 20230308T143655Z
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key: MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
Accept-Encoding: gzip

2023/03/08 09:36:55 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:55 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:36:55 DEBUG : HTTP RESPONSE (req 0xc00084c800)
2023/03/08 09:36:55 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Wed, 08 Mar 2023 14:36:57 GMT
Etag: "c24c827c73e69d2432111f75740d59d3"
Server: AmazonS3
X-Amz-Id-2: vbTYXgDT6I6mLsoVoLUQ5oIeY+rNM/+qqkFB2lspYozcO7X4pqP3Tu9nHEMlw3k9h8iAH5REsKs=
X-Amz-Request-Id: 34WGC1GP0410VMHR
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==

2023/03/08 09:36:55 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:36:55 DEBUG : file.ext: multipart upload starting chunk 6 size 10Mi offset 50Mi/100Mi
2023/03/08 09:36:56 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:56 DEBUG : HTTP REQUEST (req 0xc00084c400)
2023/03/08 09:36:56 DEBUG : PUT /ssec.and.etag/file.ext?partNumber=6&uploadId=FDcCY2scQU_RorCofQ4Hyoh3xND0B7ad387Sao49.BDyPQfK5syqUMPpam.nR.BlfnOjmvs.kq4kOV8BpwMYwg-- HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6741.f793faae4.fix-s3-sse-multipart
Content-Length: 10485760
Authorization: XXXX
Content-Md5: sVeZXCWnR0VxNqJwBGb+Yw==
Expect: 100-Continue
X-Amz-Content-Sha256: 1aee3c6b41000a2004496c4578785ea57748a301294e212f87467d303b40e827
X-Amz-Date: 20230308T143656Z
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key: MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
Accept-Encoding: gzip

2023/03/08 09:36:56 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:56 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:36:56 DEBUG : HTTP RESPONSE (req 0xc00084c400)
2023/03/08 09:36:56 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Wed, 08 Mar 2023 14:36:58 GMT
Etag: "b8e84128e190eec22ce3c1308bc158ee"
Server: AmazonS3
X-Amz-Id-2: h/2zjZZRMG0a4XYBfC61fjgCGT/8bErS24eNzDHMTHx3v+VduWP2sLZMS39IlS0LWBOGRXDzMBI=
X-Amz-Request-Id: KM844S5SP26VYVD5
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==

2023/03/08 09:36:56 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:36:56 DEBUG : file.ext: multipart upload starting chunk 7 size 10Mi offset 60Mi/100Mi
2023/03/08 09:36:56 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:56 DEBUG : HTTP REQUEST (req 0xc000a7a100)
2023/03/08 09:36:56 DEBUG : PUT /ssec.and.etag/file.ext?partNumber=7&uploadId=FDcCY2scQU_RorCofQ4Hyoh3xND0B7ad387Sao49.BDyPQfK5syqUMPpam.nR.BlfnOjmvs.kq4kOV8BpwMYwg-- HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6741.f793faae4.fix-s3-sse-multipart
Content-Length: 10485760
Authorization: XXXX
Content-Md5: 8anVkoNciJKmmZfdYrWoQQ==
Expect: 100-Continue
X-Amz-Content-Sha256: 238fae0cd097181555bb5109c910647b06af860073072052f509a5828404b68b
X-Amz-Date: 20230308T143656Z
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key: MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
Accept-Encoding: gzip

2023/03/08 09:36:56 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:57 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:36:57 DEBUG : HTTP RESPONSE (req 0xc000a7a100)
2023/03/08 09:36:57 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Wed, 08 Mar 2023 14:36:58 GMT
Etag: "220bb561aa9730f16f4b8074bd9eb549"
Server: AmazonS3
X-Amz-Id-2: ZXhmQ781+dg/cTl0tFzO8+9PEBHrAXuiSkZXycyEFXcQDwiRnE/bbhfeQhWWR0MTHi6v4uGlty8=
X-Amz-Request-Id: KM83PT9P95P0JY4K
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==

2023/03/08 09:36:57 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:36:57 DEBUG : file.ext: multipart upload starting chunk 8 size 10Mi offset 70Mi/100Mi
2023/03/08 09:36:57 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:57 DEBUG : HTTP REQUEST (req 0xc000a7a700)
2023/03/08 09:36:57 DEBUG : PUT /ssec.and.etag/file.ext?partNumber=8&uploadId=FDcCY2scQU_RorCofQ4Hyoh3xND0B7ad387Sao49.BDyPQfK5syqUMPpam.nR.BlfnOjmvs.kq4kOV8BpwMYwg-- HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6741.f793faae4.fix-s3-sse-multipart
Content-Length: 10485760
Authorization: XXXX
Content-Md5: wSrOoZjwyJMAe4Pxfd7dFg==
Expect: 100-Continue
X-Amz-Content-Sha256: 03bdeb2a0abd5fb5587812eb0d90b863682218713f5c10ee1b530824bb9f0256
X-Amz-Date: 20230308T143657Z
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key: MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
Accept-Encoding: gzip

2023/03/08 09:36:57 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:58 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:36:58 DEBUG : HTTP RESPONSE (req 0xc000a7a700)
2023/03/08 09:36:58 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Wed, 08 Mar 2023 14:36:59 GMT
Etag: "e615b5263a49cde19c6f9868fb5c3441"
Server: AmazonS3
X-Amz-Id-2: zq2X9fUBh2T2N+cFjCcMC6Kp5CPMi91ZXhzfhFDZo2W6OvGCmsNt9Mvsz40JRC5AMWPKdCtSfzc=
X-Amz-Request-Id: QMGTPQ8HV8JWDG6Z
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==

2023/03/08 09:36:58 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:36:58 DEBUG : file.ext: multipart upload starting chunk 9 size 10Mi offset 80Mi/100Mi
2023/03/08 09:36:58 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:58 DEBUG : HTTP REQUEST (req 0xc0008acc00)
2023/03/08 09:36:58 DEBUG : PUT /ssec.and.etag/file.ext?partNumber=9&uploadId=FDcCY2scQU_RorCofQ4Hyoh3xND0B7ad387Sao49.BDyPQfK5syqUMPpam.nR.BlfnOjmvs.kq4kOV8BpwMYwg-- HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6741.f793faae4.fix-s3-sse-multipart
Content-Length: 10485760
Authorization: XXXX
Content-Md5: e8eTA22VCBMBJIOKJXAKWw==
Expect: 100-Continue
X-Amz-Content-Sha256: 9dbdce8c67deea24a10b88b9ab1f2d4929feb7ea49c283d33e5ac936625877ca
X-Amz-Date: 20230308T143658Z
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key: MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
Accept-Encoding: gzip

2023/03/08 09:36:58 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:59 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:36:59 DEBUG : HTTP RESPONSE (req 0xc0008acc00)
2023/03/08 09:36:59 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Wed, 08 Mar 2023 14:37:00 GMT
Etag: "46e790723154d9b45821f6a1b6b7c9c2"
Server: AmazonS3
X-Amz-Id-2: eXGmJUKqEtRUq4tGu3kDJlyNhgnOud2pmGcD2msCnO9g/4QEHyT4iInkkyeLmXP8MRSB0Ll72Uk=
X-Amz-Request-Id: 1WXNNQXM34HRPRPG
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==

2023/03/08 09:36:59 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:36:59 DEBUG : file.ext: multipart upload starting chunk 10 size 10Mi offset 90Mi/100Mi
2023/03/08 09:36:59 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:59 DEBUG : HTTP REQUEST (req 0xc000a7a000)
2023/03/08 09:36:59 DEBUG : PUT /ssec.and.etag/file.ext?partNumber=10&uploadId=FDcCY2scQU_RorCofQ4Hyoh3xND0B7ad387Sao49.BDyPQfK5syqUMPpam.nR.BlfnOjmvs.kq4kOV8BpwMYwg-- HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6741.f793faae4.fix-s3-sse-multipart
Content-Length: 10485760
Authorization: XXXX
Content-Md5: hpWEdc4fUwNiTObOA/xFSQ==
Expect: 100-Continue
X-Amz-Content-Sha256: a279cf7e76603504fbf16eee7103191d8fdc3d5c36c576b4123e0ad6fb40ea62
X-Amz-Date: 20230308T143659Z
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key: MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
Accept-Encoding: gzip

2023/03/08 09:36:59 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:59 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:36:59 DEBUG : HTTP RESPONSE (req 0xc000a7a000)
2023/03/08 09:36:59 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Wed, 08 Mar 2023 14:37:01 GMT
Etag: "b8d1cc8eda9c0104d4575c6dd774aec6"
Server: AmazonS3
X-Amz-Id-2: XwQiH3STPy7YxtFRCN04on6N7LgZss5mofIhh0rP+AxzS0nlpNR/1EtDl2nxEqYrgJBIucPgi2c=
X-Amz-Request-Id: AQ5Y5XRFWK4YS658
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==

2023/03/08 09:36:59 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:36:59 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:59 DEBUG : HTTP REQUEST (req 0xc0008acc00)
2023/03/08 09:36:59 DEBUG : POST /ssec.and.etag/file.ext?uploadId=FDcCY2scQU_RorCofQ4Hyoh3xND0B7ad387Sao49.BDyPQfK5syqUMPpam.nR.BlfnOjmvs.kq4kOV8BpwMYwg-- HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6741.f793faae4.fix-s3-sse-multipart
Content-Length: 1040
Authorization: XXXX
X-Amz-Content-Sha256: 7a683b9981004bacdcc50a339a6458c8cc54e172cc0f12666ecd72b47774dfcd
X-Amz-Date: 20230308T143659Z
Accept-Encoding: gzip

2023/03/08 09:36:59 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:59 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:36:59 DEBUG : HTTP RESPONSE (req 0xc0008acc00)
2023/03/08 09:37:00 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Wed, 08 Mar 2023 14:37:01 GMT
Server: AmazonS3
X-Amz-Id-2: +E38CKpH7XAbllZQ9jUMt/mDaA/Hiamz4XwroY7Qx3keLuQIFGSrXzhdZArhlAjQ/M+To8J9mJo=
X-Amz-Request-Id: AQ5VXBMFCFSC6KBM
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256

152
<?xml version="1.0" encoding="UTF-8"?>

<CompleteMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Location>https://s3.us-east-1.amazonaws.com/ssec.and.etag/file.ext</Location><Bucket>ssec.and.etag</Bucket><Key>file.ext</Key><ETag>&quot;0b844664e3e49247b7ed8ae0871ae399-10&quot;</ETag></CompleteMultipartUploadResult>
0

2023/03/08 09:37:00 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:37:00 ERROR : file.ext: Failed to copy: multipart upload corrupted: Etag differ: expecting 5fa6d124612290af0c85759cb50ce07a-10 but got 0b844664e3e49247b7ed8ae0871ae399-10
2023/03/08 09:37:00 ERROR : Attempt 1/1 failed with 1 errors and: multipart upload corrupted: Etag differ: expecting 5fa6d124612290af0c85759cb50ce07a-10 but got 0b844664e3e49247b7ed8ae0871ae399-10
2023/03/08 09:37:00 INFO  : 
Transferred:   	      100 MiB / 100 MiB, 100%, 12.499 MiB/s, ETA 0s
Errors:                 1 (retrying may help)
Elapsed time:         8.5s

2023/03/08 09:37:00 DEBUG : 8 go routines active
2023/03/08 09:37:00 Failed to copy: multipart upload corrupted: Etag differ: expecting 5fa6d124612290af0c85759cb50ce07a-10 but got 0b844664e3e49247b7ed8ae0871ae399-10