In some cases, rclone does not use ETag to verify files

Help and Support
1

What is the problem you are having with rclone?

after file transfer completes, force rclone to compare rclone calculated ETag against provider calculated ETag.

from the log, looks like wasabi returns the Etag, yet rclone does not use it?

<CompleteMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Location>https://azork.s3.us-east-2.wasabisys.com/file.ext</Location><Bucket>azork</Bucket><Key>file.ext</Key>
<ETag>&quot;195c549125ec2fe75e9f3e8444b4a98c-2&quot;</ETag></CompleteMultipartUploadResult>

i want this

DEBUG : file.ext: Multipart upload Etag: 195c549125ec2fe75e9f3e8444b4a98c-2 OK
DEBUG : file.ext: md5 = 7246aff55b4ba5cdd90912d8819d5a0f OK
INFO  : file.ext: Copied (new)

instead i get this

DEBUG : file.ext: md5 = 7246aff55b4ba5cdd90912d8819d5a0f OK
INFO  : file.ext: Copied (new)

note: the issue occurs when using this locked down bucket policy and as a result, have to use --s3-no-head

{  "Version": "2012-10-17",
   "Statement": [{
      "Effect": "Allow",
      "Principal": {"AWS": "arn:aws:iam::redacted:user/zork"},
      "Action": ["s3:ListBucket", "s3:PutObject"],
      "Resource": ["arn:aws:s3:::azork/*", "arn:aws:s3:::azork"]}]
}

Run the command 'rclone version' and share the full output of the command.

rclone v1.61.1
- os/version: ubuntu 22.04 (64 bit)
- os/kernel: 5.15.79.1-microsoft-standard-WSL2 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.19.4
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

wasabi, s3 provider

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone check ./source azork: --size-only --one-way --missing-on-dst=missing-on-dst.txt -vv 
rclone copy ./source azork: --s3-use-multipart-etag=true --s3-no-head --s3-upload-cutoff=0 --s3-chunk-size=19M --files-from=missing-on-dst.txt --dump=headers,responses

The rclone config contents with secrets removed.

[wasabi_azork_remote]
type = s3
provider = Wasabi
access_key_id = redacted
secret_access_key = redacted
endpoint = s3.us-east-2.wasabisys.com

[azork]
type = alias
remote = wasabi_azork_remote:azork

A log from the command with the -vv flag


rclone copy ./source azork: --files-from=missing-on-dst.txt --s3-no-head --s3-upload-cutoff=0 --s3-chunk-size=19M --dump=headers,responses -vv
2023/02/12 10:51:55 DEBUG : rclone: Version "v1.61.1" starting with parameters ["rclone" "copy" "./source" "azork:" "-vv" "--retries=1" "--files-from=missing-on-dst.txt" "--s3-no-head" "--s3-upload-cutoff=0" "--s3-chunk-size=19M" "--dump=headers,responses" "--s3-use-multipart-etag=true"]
2023/02/12 10:51:55 DEBUG : Creating backend with remote "./source"
2023/02/12 10:51:55 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2023/02/12 10:51:55 DEBUG : fs cache: renaming cache item "./source" to be canonical "/root/rclone/scripts/source"
2023/02/12 10:51:55 DEBUG : Creating backend with remote "azork:"
2023/02/12 10:51:55 DEBUG : Creating backend with remote "wasabi_azork_remote:azork"
2023/02/12 10:51:55 DEBUG : wasabi_azork_remote: detected overridden config - adding "{EWbYf}" suffix to name
2023/02/12 10:51:55 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2023/02/12 10:51:55 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2023/02/12 10:51:55 DEBUG : fs cache: renaming cache item "wasabi_azork_remote:azork" to be canonical "wasabi_azork_remote{EWbYf}:azork"
2023/02/12 10:51:55 DEBUG : fs cache: renaming cache item "azork:" to be canonical "wasabi_azork_remote{EWbYf}:azork"
2023/02/12 10:51:55 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/12 10:51:55 DEBUG : HTTP REQUEST (req 0xc0008dab00)
2023/02/12 10:51:55 DEBUG : GET /?delimiter=%2F&encoding-type=url&list-type=2&max-keys=1000&prefix= HTTP/1.1
Host: azork.s3.us-east-2.wasabisys.com
User-Agent: rclone/v1.61.1
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20230212T155155Z
Accept-Encoding: gzip

2023/02/12 10:51:55 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/12 10:51:56 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/12 10:51:56 DEBUG : HTTP RESPONSE (req 0xc0008dab00)
2023/02/12 10:51:56 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Sun, 12 Feb 2023 15:51:56 GMT
Server: WasabiS3/7.10.1198-2022-12-14-39a7a2e69e (B33-U25)
X-Amz-Bucket-Region: us-east-2
X-Amz-Id-2: X4qgZ7hl17jxppDU8TwveloepcI5HOS/35d52ERkD+ZaapD2I54WoLiD0xrrmUknSlnKo+nXhFrm
X-Amz-Request-Id: 5FB66F630CB45F6D:A

124
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>azork</Name><Prefix></Prefix><KeyCount>2</KeyCount><MaxKeys>1000</MaxKeys><Delimiter>/</Delimiter><EncodingType>url</EncodingType><IsTruncated>false</IsTruncated></ListBucketResult>
0

2023/02/12 10:51:56 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/12 10:51:56 DEBUG : S3 bucket azork: Waiting for checks to finish
2023/02/12 10:51:56 DEBUG : S3 bucket azork: Waiting for transfers to finish
2023/02/12 10:51:56 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/12 10:51:56 DEBUG : HTTP REQUEST (req 0xc0008dac00)
2023/02/12 10:51:56 DEBUG : POST /file.ext?uploads= HTTP/1.1
Host: azork.s3.us-east-2.wasabisys.com
User-Agent: rclone/v1.61.1
Content-Length: 0
Authorization: XXXX
Content-Type: application/octet-stream
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20230212T155156Z
X-Amz-Meta-Md5chksum: ckav9VtLpc3ZCRLYgZ1aDw==
X-Amz-Meta-Mtime: 1676217114.643056688
Accept-Encoding: gzip

2023/02/12 10:51:56 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/12 10:51:56 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/12 10:51:56 DEBUG : HTTP RESPONSE (req 0xc0008dac00)
2023/02/12 10:51:56 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Date: Sun, 12 Feb 2023 15:51:56 GMT
Server: WasabiS3/7.10.1198-2022-12-14-39a7a2e69e (B33-U25)
X-Amz-Id-2: L6JhR7W0Ia+5VO8AifCM7SELoTQk5dzmoMrqgTgqu247vlGGFTq5D74c25c/TogRGe0xUZKcSQdn
X-Amz-Request-Id: 16C09B78C9A0714D:A

154
<?xml version="1.0" encoding="UTF-8"?>
<InitiateMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Bucket>azork</Bucket><Key>file.ext</Key><UploadId>u3_2_FKr98usheWHYacDfnD1gHfeIeltoJhNX18KYYbQXXwsy9cE_e8UTVLNB1wtXcnIRuvUTQM5v8Cib2g8nNByc9xs2mfU6GFUqaYxHoG0DBDYkSTDJCBCF9bulvGb</UploadId></InitiateMultipartUploadResult>
0

2023/02/12 10:51:56 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/12 10:51:56 DEBUG : file.ext: multipart upload starting chunk 1 size 19Mi offset 0/20Mi
2023/02/12 10:51:56 DEBUG : file.ext: multipart upload starting chunk 2 size 1Mi offset 19Mi/20Mi
2023/02/12 10:51:56 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/12 10:51:56 DEBUG : HTTP REQUEST (req 0xc0008dab00)
2023/02/12 10:51:56 DEBUG : PUT /file.ext?partNumber=2&uploadId=u3_2_FKr98usheWHYacDfnD1gHfeIeltoJhNX18KYYbQXXwsy9cE_e8UTVLNB1wtXcnIRuvUTQM5v8Cib2g8nNByc9xs2mfU6GFUqaYxHoG0DBDYkSTDJCBCF9bulvGb HTTP/1.1
Host: azork.s3.us-east-2.wasabisys.com
User-Agent: rclone/v1.61.1
Content-Length: 1048576
Authorization: XXXX
Content-Md5: aUHYpoSioPsWAqy4QOT46Q==
X-Amz-Content-Sha256: 66de6f9b5138899a8c8decc8f1cf1cbb99fa8e2c52b2a0e6b45eeeae2dffbf1e
X-Amz-Date: 20230212T155156Z
Accept-Encoding: gzip

2023/02/12 10:51:56 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/12 10:51:56 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/12 10:51:56 DEBUG : HTTP REQUEST (req 0xc0006f0200)
2023/02/12 10:51:56 DEBUG : PUT /file.ext?partNumber=1&uploadId=u3_2_FKr98usheWHYacDfnD1gHfeIeltoJhNX18KYYbQXXwsy9cE_e8UTVLNB1wtXcnIRuvUTQM5v8Cib2g8nNByc9xs2mfU6GFUqaYxHoG0DBDYkSTDJCBCF9bulvGb HTTP/1.1
Host: azork.s3.us-east-2.wasabisys.com
User-Agent: rclone/v1.61.1
Content-Length: 19922944
Authorization: XXXX
Content-Md5: n/jrKfsFqxQ2DIIWZVnx+g==
Expect: 100-Continue
X-Amz-Content-Sha256: 0f033eb7f8431b96208e604338ed327d2c5f398908b6d21cd3e6fb6e4009c363
X-Amz-Date: 20230212T155156Z
Accept-Encoding: gzip

2023/02/12 10:51:56 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/12 10:51:56 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/12 10:51:56 DEBUG : HTTP RESPONSE (req 0xc0008dab00)
2023/02/12 10:51:56 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Sun, 12 Feb 2023 15:51:56 GMT
Etag: "6941d8a684a2a0fb1602acb840e4f8e9"
Server: WasabiS3/7.10.1198-2022-12-14-39a7a2e69e (B33-U25)
X-Amz-Id-2: e6wV/ViO9CtVj2n7hMyGYcgE21T9h15AhRV4r3AfHLWTSvElZENqqJkXNNgnANVqG2l7y3KLqMul
X-Amz-Request-Id: 2B942E69FCD0A8CD:A

2023/02/12 10:51:56 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/12 10:51:57 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/12 10:51:57 DEBUG : HTTP RESPONSE (req 0xc0006f0200)
2023/02/12 10:51:57 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Sun, 12 Feb 2023 15:51:57 GMT
Etag: "9ff8eb29fb05ab14360c82166559f1fa"
Server: WasabiS3/7.10.1198-2022-12-14-39a7a2e69e (XB27-U40)
X-Amz-Id-2: lxaRY6gsBItfbColH33BMqFQJQBLgBQy2YTkg2FNXW9j71GAh6udosnblCatyaI6/RaT7/4tZwxV
X-Amz-Request-Id: 7C40CBE5BA852FE4:A

2023/02/12 10:51:57 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/12 10:51:57 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/12 10:51:57 DEBUG : HTTP REQUEST (req 0xc0008dad00)
2023/02/12 10:51:57 DEBUG : POST /file.ext?uploadId=u3_2_FKr98usheWHYacDfnD1gHfeIeltoJhNX18KYYbQXXwsy9cE_e8UTVLNB1wtXcnIRuvUTQM5v8Cib2g8nNByc9xs2mfU6GFUqaYxHoG0DBDYkSTDJCBCF9bulvGb HTTP/1.1
Host: azork.s3.us-east-2.wasabisys.com
User-Agent: rclone/v1.61.1
Content-Length: 287
Authorization: XXXX
X-Amz-Content-Sha256: d204a9abf0a8b8d18bd291b0675bdfd86c490a4520ec7c40a1dd66d6cdc297e7
X-Amz-Date: 20230212T155157Z
Accept-Encoding: gzip

2023/02/12 10:51:57 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/12 10:51:57 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/12 10:51:57 DEBUG : HTTP RESPONSE (req 0xc0008dad00)
2023/02/12 10:51:57 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Sun, 12 Feb 2023 15:51:57 GMT
Server: WasabiS3/7.10.1198-2022-12-14-39a7a2e69e (XB27-U40)
X-Amz-Id-2: 00AA7cnTDV/PHuAAMYxow7mqzA9WbrveA6qN5Yc14L/a3eKC+GAl/gZY8Rr4DumgfA2xjzggQYIM
X-Amz-Request-Id: 622C041D60C48EF7:A
X-Amz-Version-Id: 001676217116418210501-ugBQes-Rwr

140
<?xml version="1.0" encoding="UTF-8"?>
<CompleteMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Location>https://azork.s3.us-east-2.wasabisys.com/file.ext</Location><Bucket>azork</Bucket><Key>file.ext</Key><ETag>&quot;195c549125ec2fe75e9f3e8444b4a98c-2&quot;</ETag></CompleteMultipartUploadResult>
0

2023/02/12 10:51:57 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/12 10:51:57 DEBUG : file.ext: md5 = 7246aff55b4ba5cdd90912d8819d5a0f OK
2023/02/12 10:51:57 INFO  : file.ext: Copied (new)
2023/02/12 10:51:57 INFO  : 
Transferred:   	       20 MiB / 20 MiB, 100%, 19.978 MiB/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:         1.5s

2023/02/12 10:51:57 DEBUG : 9 go routines active
2

So this is a side effect of --s3-no-head?

At the moment rclone heads the uploaded object to read the etag - I think you are saying rclone should use the etag in the upload complete response if using no head?

That could probably be arranged if I'm understanding you correctly!

3

great.
if provider returns ETag:
then rclone should use it, with respect to --s3-use-multipart-etag

as i understand it,

  • for multipart, x-amz-meta-md5chksum cannot be used.
  • so this log entry cannot be relied upon.
    DEBUG : file.ext: md5 = 7246aff55b4ba5cdd90912d8819d5a0f OK

in addition, i trying to create WORZ - Write Once, Read Zed.
once a file is uploaded, no way to download , no way to overwrite, no way to delete.

4

I didn't realise that the ETag came back in the final POST for the multipart upload - well spotted.

I've had a go at implementing that here - please give it a go.

v1.62.0-beta.6731.63bf57580.s3-multipart-etag on branch s3-multipart-etag (uploaded in 15-30 mins)

It isn't a good check you are right.

However multipart uploads are protected in a number of different ways - the best probably being that rclone sends the Hashes of all the individual parts in a list at the end in the last POST request so the provider should be doing the equivalent of that check when rclone requests the multipart upload to be completed.

Adding this check when using --s3-no-head is very nice though and a tiny bit more certainty that things are correct :slight_smile:

5

i am trying to achieve --bullet-proof
This would then enable a bullet proof end to end upload check for multipart objects

the beta worked!

2023/02/13 09:13:37 DEBUG : file.ext: Multipart upload Etag: 195c549125ec2fe75e9f3e8444b4a98c-2 OK
2023/02/13 09:13:37 DEBUG : file.ext: md5 = 7246aff55b4ba5cdd90912d8819d5a0f OK
2023/02/13 09:13:37 INFO  : file.ext: Copied (new)

and here is the full log

rclone copy ./source azork: --s3-no-head --s3-upload-cutoff=0 --s3-chunk-size=19M -vv --retries=1 --files-from=missing-on-dst.txt --dump=headers,responses
2023/02/13 09:13:36 DEBUG : rclone: Version "v1.62.0-beta.6731.63bf57580.s3-multipart-etag" starting with parameters ["./rclone" "copy" "./source" "azork:" "--s3-no-head" "--s3-upload-cutoff=0" "--s3-chunk-size=19M" "-vv" "--retries=1" "--files-from=missing-on-dst.txt" "--dump=headers,responses"]
2023/02/13 09:13:36 DEBUG : Creating backend with remote "./source"
2023/02/13 09:13:36 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2023/02/13 09:13:36 DEBUG : fs cache: renaming cache item "./source" to be canonical "/root/rclone/scripts/minimal/source"
2023/02/13 09:13:36 DEBUG : Creating backend with remote "azork:"
2023/02/13 09:13:36 DEBUG : Creating backend with remote "wasabi_azork_remote:azork"
2023/02/13 09:13:36 DEBUG : wasabi_azork_remote: detected overridden config - adding "{D0imL}" suffix to name
2023/02/13 09:13:36 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2023/02/13 09:13:36 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2023/02/13 09:13:36 DEBUG : fs cache: renaming cache item "wasabi_azork_remote:azork" to be canonical "wasabi_azork_remote{D0imL}:azork"
2023/02/13 09:13:36 DEBUG : fs cache: renaming cache item "azork:" to be canonical "wasabi_azork_remote{D0imL}:azork"
2023/02/13 09:13:36 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/13 09:13:36 DEBUG : HTTP REQUEST (req 0xc0009b8000)
2023/02/13 09:13:36 DEBUG : GET /?delimiter=%2F&encoding-type=url&list-type=2&max-keys=1000&prefix= HTTP/1.1
Host: azork.s3.us-east-2.wasabisys.com
User-Agent: rclone/v1.62.0-beta.6731.63bf57580.s3-multipart-etag
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20230213T141336Z
Accept-Encoding: gzip

2023/02/13 09:13:36 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/13 09:13:36 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/13 09:13:36 DEBUG : HTTP RESPONSE (req 0xc0009b8000)
2023/02/13 09:13:36 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Mon, 13 Feb 2023 14:13:36 GMT
Server: WasabiS3/7.10.1198-2022-12-14-39a7a2e69e (XB27-U41)
X-Amz-Bucket-Region: us-east-2
X-Amz-Id-2: 1s95hhGXzJKysk/ZnyJid1zDIZInZOgnFTYYhgrkx/4j7AqY21il6GUxqkQW9FbEBm5SDMCuGFvu
X-Amz-Request-Id: 545E413EDA79C104:A

124
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>azork</Name><Prefix></Prefix><KeyCount>2</KeyCount><MaxKeys>1000</MaxKeys><Delimiter>/</Delimiter><EncodingType>url</EncodingType><IsTruncated>false</IsTruncated></ListBucketResult>
0

2023/02/13 09:13:36 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/13 09:13:36 DEBUG : S3 bucket azork: Waiting for checks to finish
2023/02/13 09:13:36 DEBUG : S3 bucket azork: Waiting for transfers to finish
2023/02/13 09:13:36 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/13 09:13:36 DEBUG : HTTP REQUEST (req 0xc0009b8300)
2023/02/13 09:13:36 DEBUG : POST /file.ext?uploads= HTTP/1.1
Host: azork.s3.us-east-2.wasabisys.com
User-Agent: rclone/v1.62.0-beta.6731.63bf57580.s3-multipart-etag
Content-Length: 0
Authorization: XXXX
Content-Type: application/octet-stream
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20230213T141336Z
X-Amz-Meta-Md5chksum: ckav9VtLpc3ZCRLYgZ1aDw==
X-Amz-Meta-Mtime: 1676297614.651237148
Accept-Encoding: gzip

2023/02/13 09:13:36 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/13 09:13:36 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/13 09:13:36 DEBUG : HTTP RESPONSE (req 0xc0009b8300)
2023/02/13 09:13:36 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Date: Mon, 13 Feb 2023 14:13:36 GMT
Server: WasabiS3/7.10.1198-2022-12-14-39a7a2e69e (XB27-U41)
X-Amz-Id-2: xM7ZDigOK5WERgMqaQrD6yeX8EagoLQMSWMXQ8S5FFH3rCD0e1BjpPEoUXkde5gVk/DZKVFqmh7q
X-Amz-Request-Id: ACF3AAA660D96C98:A

154
<?xml version="1.0" encoding="UTF-8"?>
<InitiateMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Bucket>azork</Bucket><Key>file.ext</Key><UploadId>yUEm_XShUnHbEcPd5vrTRv9qQSSUIiWwE7omVHMoNWUT6by6jTVztGX8n_ByHXfrGO_FkM4Dszm-UHOR5hH7KlELJ_MRsHDfKgbtb3CBpk31aDbIl3tZqjhQS6bQsQYR</UploadId></InitiateMultipartUploadResult>
0

2023/02/13 09:13:36 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/13 09:13:36 DEBUG : file.ext: multipart upload starting chunk 1 size 19Mi offset 0/20Mi
2023/02/13 09:13:36 DEBUG : file.ext: multipart upload starting chunk 2 size 1Mi offset 19Mi/20Mi
2023/02/13 09:13:36 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/13 09:13:36 DEBUG : HTTP REQUEST (req 0xc000256800)
2023/02/13 09:13:36 DEBUG : PUT /file.ext?partNumber=2&uploadId=yUEm_XShUnHbEcPd5vrTRv9qQSSUIiWwE7omVHMoNWUT6by6jTVztGX8n_ByHXfrGO_FkM4Dszm-UHOR5hH7KlELJ_MRsHDfKgbtb3CBpk31aDbIl3tZqjhQS6bQsQYR HTTP/1.1
Host: azork.s3.us-east-2.wasabisys.com
User-Agent: rclone/v1.62.0-beta.6731.63bf57580.s3-multipart-etag
Content-Length: 1048576
Authorization: XXXX
Content-Md5: aUHYpoSioPsWAqy4QOT46Q==
X-Amz-Content-Sha256: 66de6f9b5138899a8c8decc8f1cf1cbb99fa8e2c52b2a0e6b45eeeae2dffbf1e
X-Amz-Date: 20230213T141336Z
Accept-Encoding: gzip

2023/02/13 09:13:36 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/13 09:13:36 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/13 09:13:36 DEBUG : HTTP REQUEST (req 0xc0009b8200)
2023/02/13 09:13:36 DEBUG : PUT /file.ext?partNumber=1&uploadId=yUEm_XShUnHbEcPd5vrTRv9qQSSUIiWwE7omVHMoNWUT6by6jTVztGX8n_ByHXfrGO_FkM4Dszm-UHOR5hH7KlELJ_MRsHDfKgbtb3CBpk31aDbIl3tZqjhQS6bQsQYR HTTP/1.1
Host: azork.s3.us-east-2.wasabisys.com
User-Agent: rclone/v1.62.0-beta.6731.63bf57580.s3-multipart-etag
Content-Length: 19922944
Authorization: XXXX
Content-Md5: n/jrKfsFqxQ2DIIWZVnx+g==
Expect: 100-Continue
X-Amz-Content-Sha256: 0f033eb7f8431b96208e604338ed327d2c5f398908b6d21cd3e6fb6e4009c363
X-Amz-Date: 20230213T141336Z
Accept-Encoding: gzip

2023/02/13 09:13:36 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/13 09:13:36 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/13 09:13:36 DEBUG : HTTP RESPONSE (req 0xc000256800)
2023/02/13 09:13:36 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Mon, 13 Feb 2023 14:13:37 GMT
Etag: "6941d8a684a2a0fb1602acb840e4f8e9"
Server: WasabiS3/7.10.1198-2022-12-14-39a7a2e69e (XB27-U41)
X-Amz-Id-2: jhIknZzO17wUJW/travDJpYn+PP735Fjl41pYk2eST4/dzZZ/ioo9/l9/c/pF//srSQBpdR4JzCW
X-Amz-Request-Id: 8879FD174587847E:A

2023/02/13 09:13:36 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/13 09:13:37 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/13 09:13:37 DEBUG : HTTP RESPONSE (req 0xc0009b8200)
2023/02/13 09:13:37 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Mon, 13 Feb 2023 14:13:38 GMT
Etag: "9ff8eb29fb05ab14360c82166559f1fa"
Server: WasabiS3/7.10.1198-2022-12-14-39a7a2e69e (B33-U25)
X-Amz-Id-2: 7dZQue+s3nxLMbSvatJCzWxytAUh3cv/NQc6NRwJIKz7zaD7zhlblz5GQ4ykatIFK+HrlCcoIVih
X-Amz-Request-Id: 7716F4824964AC56:A

2023/02/13 09:13:37 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/13 09:13:37 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/13 09:13:37 DEBUG : HTTP REQUEST (req 0xc000257500)
2023/02/13 09:13:37 DEBUG : POST /file.ext?uploadId=yUEm_XShUnHbEcPd5vrTRv9qQSSUIiWwE7omVHMoNWUT6by6jTVztGX8n_ByHXfrGO_FkM4Dszm-UHOR5hH7KlELJ_MRsHDfKgbtb3CBpk31aDbIl3tZqjhQS6bQsQYR HTTP/1.1
Host: azork.s3.us-east-2.wasabisys.com
User-Agent: rclone/v1.62.0-beta.6731.63bf57580.s3-multipart-etag
Content-Length: 287
Authorization: XXXX
X-Amz-Content-Sha256: 73cba0622eb3317477cccd8807a7cbec7330475d93b0a18e9f19b05a9d74ee02
X-Amz-Date: 20230213T141337Z
Accept-Encoding: gzip

2023/02/13 09:13:37 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/13 09:13:37 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/13 09:13:37 DEBUG : HTTP RESPONSE (req 0xc000257500)
2023/02/13 09:13:37 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Mon, 13 Feb 2023 14:13:38 GMT
Server: WasabiS3/7.10.1198-2022-12-14-39a7a2e69e (B33-U25)
X-Amz-Id-2: wJbOm4VFbTxgfmci1Uny65eYGimQihIkFBGegvHpUvpCe9tOtgv2ZVDEbxrms9O9gGLZaFS+CjM2
X-Amz-Request-Id: 2B644CD20340587B:A
X-Amz-Version-Id: 001676297616821718304-SHNDnyTeTy

140
<?xml version="1.0" encoding="UTF-8"?>
<CompleteMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Location>https://azork.s3.us-east-2.wasabisys.com/file.ext</Location><Bucket>azork</Bucket><Key>file.ext</Key><ETag>&quot;195c549125ec2fe75e9f3e8444b4a98c-2&quot;</ETag></CompleteMultipartUploadResult>
0

2023/02/13 09:13:37 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/13 09:13:37 DEBUG : file.ext: Multipart upload Etag: 195c549125ec2fe75e9f3e8444b4a98c-2 OK
2023/02/13 09:13:37 DEBUG : file.ext: md5 = 7246aff55b4ba5cdd90912d8819d5a0f OK
2023/02/13 09:13:37 INFO  : file.ext: Copied (new)
2023/02/13 09:13:37 INFO  : 
Transferred:   	       20 MiB / 20 MiB, 100%, 19.989 MiB/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:         1.8s

2023/02/13 09:13:37 DEBUG : 9 go routines active
6

when using multipart+SSE-C, wasabi and aws return the ETag, but rclone is not using it?

EDIT: at this time time, i suggest we, temporally, pause any new ETag related betas releases.

in my testing of multipart+SSE-C, i noitced that wasabi and aws return different ETag values
very confusing. so i contacted wasabi and they were able to duplicate the results.

On aws, when copying a file using SSE-C, the ETag is different than when copying without SSE-C.

SSE-C:      <ETag>&quot;84243b929bad8317967cde2bdf7b99c5-2&quot;</ETag>
No SSE-C:   <ETag>&quot;195c549125ec2fe75e9f3e8444b4a98c-2&quot;</ETag>

On wasabi, when copying a file using SSE-C, the ETag is the same as when copying without SSE-C.

SSE-C:       <ETag>&quot;195c549125ec2fe75e9f3e8444b4a98c-2&quot;</ETag>
No SSE-C:    <ETag>&quot;195c549125ec2fe75e9f3e8444b4a98c-2&quot;</ETag>
7

Great - thank you for testing and thank you for your attention to detail.

I will merge this now.

When using SSE-C the ETag in the multipart upload should be that of the encrypted data I think so rclone can't check it.

The AWS docs say

The entity tag (ETag) in the response is not the MD5 hash of the object data.

I suspect this is a bug in wasabi as it is leaking the hash of the unencrypted data.

8

in my testing, with multipart+ssec, rclone can check.

if the below looks valid, perhaps we can have rclone check the etag,
perhaps --s3-use-multipart-etag=ssec
and then i can do more testing against other s3 providers, such as wasabi

i wrote a script that

  1. from the rclone debug log, extract all the individual ETag, in the example below, 10 parts so 10 ETag values.
  2. calculate the overall ETag from all individual ETag.
  3. compare the calculated ETag against the overall ETag returned from aws

here is the output

etag parts
-------------------------------
1330e1275b556ab6702bca9438f62c15  -
ae55d3ddf52e33d45140a5be6dacb925  -
16dc956e05962b84ad9cd74a05e86797  -
64be66992a5110c4b1151a8249258a1a  -
4926df0200fe24499524176d6a85e347  -
2b6655c3506481eb1fae6b2e2e7c4b8b  -
a02e9dbd49039eaf4d6de1fddc5e1a30  -
afb7bc1f6e0c1f23671cb7116f3b0c63  -
dddf3a1ab192f26bb483a3e2778bab13  -
adb8b2b761640418856853f3810ac45a  -
-------------------------------

etag_from_aws   = c68db040f8a36c164259bcca40c36410-10
etag_calculated = c68db040f8a36c164259bcca40c36410-10
The ETags match

now, here is the script. pass the path of the rclone log file as the first argument.

cat $1 | grep -oP 'Etag: "\K[^"]+' | sed 's/$/  -/' > etags_from_parts.txt
cat $1 | grep -oP '<ETag>&quot;\K[^&quot]+' > etag_from_aws.txt

etag_from_aws=$(cat etag_from_aws.txt)
temp1=$(xxd -r -p  etags_from_parts.txt | md5sum)
temp2=$(echo $temp1 | tail -c+1 | head -c32)
partnum=$(wc -l < etags_from_parts.txt)
etag_calculated="${temp2}-${partnum}"

echo;	echo etag parts ;echo "-------------------------------"
cat etags_from_parts.txt
echo "-------------------------------"; echo 

echo "etag_from_aws   = ${etag_from_aws}"
echo "etag_calculated = ${etag_calculated}"

if [ $etag_from_aws = $etag_calculated ]; 
then  echo 'The ETags match'
else  echo 'The ETags do not match'
fi

and here is the debug log,

2023/02/15 16:53:16 DEBUG : rclone: Version "v1.62.0-beta.6731.63bf57580.s3-multipart-etag" starting with parameters ["./rclone" "copy" "./source" "aws_ssec.and.etag:" "--s3-no-head" "--s3-upload-concurrency=1" "--s3-upload-cutoff=0" "--s3-chunk-size=10M" "-vv" "--retries=1" "--files-from=missing-on-dst.txt" "--dump=headers,responses" "--log-file=debug_aws_ssec.txt"]
2023/02/15 16:53:16 DEBUG : Creating backend with remote "./source"
2023/02/15 16:53:16 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2023/02/15 16:53:16 DEBUG : fs cache: renaming cache item "./source" to be canonical "/root/rclone/scripts/minimal/source"
2023/02/15 16:53:16 DEBUG : Creating backend with remote "aws_ssec.and.etag:"
2023/02/15 16:53:16 DEBUG : Creating backend with remote "aws_ssec.and.etag_remote:ssec.and.etag"
2023/02/15 16:53:16 DEBUG : aws_ssec.and.etag_remote: detected overridden config - adding "{bsCrg}" suffix to name
2023/02/15 16:53:16 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2023/02/15 16:53:16 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2023/02/15 16:53:16 DEBUG : fs cache: renaming cache item "aws_ssec.and.etag_remote:ssec.and.etag" to be canonical "aws_ssec.and.etag_remote{bsCrg}:ssec.and.etag"
2023/02/15 16:53:16 DEBUG : fs cache: renaming cache item "aws_ssec.and.etag:" to be canonical "aws_ssec.and.etag_remote{bsCrg}:ssec.and.etag"
2023/02/15 16:53:16 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:16 DEBUG : HTTP REQUEST (req 0xc000259000)
2023/02/15 16:53:16 DEBUG : GET /ssec.and.etag?delimiter=%2F&encoding-type=url&list-type=2&max-keys=1000&prefix= HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6731.63bf57580.s3-multipart-etag
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20230215T215316Z
Accept-Encoding: gzip

2023/02/15 16:53:16 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:16 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:16 DEBUG : HTTP RESPONSE (req 0xc000259000)
2023/02/15 16:53:16 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Wed, 15 Feb 2023 21:53:18 GMT
Server: AmazonS3
X-Amz-Bucket-Region: us-east-1
X-Amz-Id-2: JOVw+xqzartxtWtqo0XCuv6ySfyG5YY6vn6k59kadv6aAGacq7iKKQp1AbXRUSQg95zbK8Aw5qo=
X-Amz-Request-Id: B4SFV1X36Y230X9N

12c
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>ssec.and.etag</Name><Prefix></Prefix><KeyCount>0</KeyCount><MaxKeys>1000</MaxKeys><Delimiter>/</Delimiter><EncodingType>url</EncodingType><IsTruncated>false</IsTruncated></ListBucketResult>
0

2023/02/15 16:53:16 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:16 DEBUG : S3 bucket ssec.and.etag: Waiting for checks to finish
2023/02/15 16:53:16 DEBUG : S3 bucket ssec.and.etag: Waiting for transfers to finish
2023/02/15 16:53:16 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:16 DEBUG : HTTP REQUEST (req 0xc00084b300)
2023/02/15 16:53:16 DEBUG : POST /ssec.and.etag/file.ext?uploads= HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6731.63bf57580.s3-multipart-etag
Content-Length: 0
Authorization: XXXX
Content-Type: application/octet-stream
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20230215T215316Z
X-Amz-Meta-Md5chksum: F/GNp36IlQwzm5A8zJC66g==
X-Amz-Meta-Mtime: 1676496659.931
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key: MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
X-Amz-Storage-Class: STANDARD
Accept-Encoding: gzip

2023/02/15 16:53:16 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:16 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:16 DEBUG : HTTP RESPONSE (req 0xc00084b300)
2023/02/15 16:53:16 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Date: Wed, 15 Feb 2023 21:53:18 GMT
Server: AmazonS3
X-Amz-Id-2: p524Cdfmo7IqJdQ6QiTelJUlRSvNG52xoSLhIfOJCqNZALnopbIMVJZPBCfC1STA08MIzqjtLbY=
X-Amz-Request-Id: B4S1B3KZYDG7S49B
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==

134
<?xml version="1.0" encoding="UTF-8"?>
<InitiateMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Bucket>ssec.and.etag</Bucket><Key>file.ext</Key><UploadId>qQeIrK7W.SsqPZz4L_htKuZDIntI0zyGiQkthh7hE_VbbtULdwClSq5rgWllCiFjP2v1gr_K.Cb8GW57M3WdbQ--</UploadId></InitiateMultipartUploadResult>
0

2023/02/15 16:53:16 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:16 DEBUG : file.ext: multipart upload starting chunk 1 size 10Mi offset 0/100Mi
2023/02/15 16:53:16 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:16 DEBUG : HTTP REQUEST (req 0xc000258600)
2023/02/15 16:53:16 DEBUG : PUT /ssec.and.etag/file.ext?partNumber=1&uploadId=qQeIrK7W.SsqPZz4L_htKuZDIntI0zyGiQkthh7hE_VbbtULdwClSq5rgWllCiFjP2v1gr_K.Cb8GW57M3WdbQ-- HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6731.63bf57580.s3-multipart-etag
Content-Length: 10485760
Authorization: XXXX
Content-Md5: lkipYzu2pL322EXK7RTjZg==
Expect: 100-Continue
X-Amz-Content-Sha256: 2466895a4ae323cdda7c41cad749ac3e4032175594e7c39ede3128a213150e96
X-Amz-Date: 20230215T215316Z
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key: MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
Accept-Encoding: gzip

2023/02/15 16:53:16 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:17 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:17 DEBUG : HTTP RESPONSE (req 0xc000258600)
2023/02/15 16:53:17 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Wed, 15 Feb 2023 21:53:18 GMT
Etag: "1330e1275b556ab6702bca9438f62c15"
Server: AmazonS3
X-Amz-Id-2: FisMLP0rbEZDCzc014QH8JZ7zz9ACSqPMeEiNYljBKj+8VcdgqZA7wkszvmEP7I1Aj4uG5TGHzA=
X-Amz-Request-Id: B4SEHY70HQTX435R
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==

2023/02/15 16:53:17 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:17 DEBUG : file.ext: multipart upload starting chunk 2 size 10Mi offset 10Mi/100Mi
2023/02/15 16:53:17 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:17 DEBUG : HTTP REQUEST (req 0xc000259300)
2023/02/15 16:53:17 DEBUG : PUT /ssec.and.etag/file.ext?partNumber=2&uploadId=qQeIrK7W.SsqPZz4L_htKuZDIntI0zyGiQkthh7hE_VbbtULdwClSq5rgWllCiFjP2v1gr_K.Cb8GW57M3WdbQ-- HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6731.63bf57580.s3-multipart-etag
Content-Length: 10485760
Authorization: XXXX
Content-Md5: FcJqqZe9uR+rzQBZPw2IwA==
Expect: 100-Continue
X-Amz-Content-Sha256: 2eec1b74ef5a55e40ec1329794275b6197a48175512460e4df9910a982d040e8
X-Amz-Date: 20230215T215317Z
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key: MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
Accept-Encoding: gzip

2023/02/15 16:53:17 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:17 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:17 DEBUG : HTTP RESPONSE (req 0xc000259300)
2023/02/15 16:53:17 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Wed, 15 Feb 2023 21:53:19 GMT
Etag: "ae55d3ddf52e33d45140a5be6dacb925"
Server: AmazonS3
X-Amz-Id-2: chAaKXGwpvLjBcTk1xvQAPSvZqCWbWOhHunEAVUkv2o0kAJcgbZMGj4LXxW6K7xoYaHuP4sbn2o=
X-Amz-Request-Id: 1BMKXWE7N7F9H6ZH
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==

2023/02/15 16:53:17 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:17 DEBUG : file.ext: multipart upload starting chunk 3 size 10Mi offset 20Mi/100Mi
2023/02/15 16:53:17 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:17 DEBUG : HTTP REQUEST (req 0xc000258600)
2023/02/15 16:53:17 DEBUG : PUT /ssec.and.etag/file.ext?partNumber=3&uploadId=qQeIrK7W.SsqPZz4L_htKuZDIntI0zyGiQkthh7hE_VbbtULdwClSq5rgWllCiFjP2v1gr_K.Cb8GW57M3WdbQ-- HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6731.63bf57580.s3-multipart-etag
Content-Length: 10485760
Authorization: XXXX
Content-Md5: B0FYoWA53eaRcsAGB6oNfg==
Expect: 100-Continue
X-Amz-Content-Sha256: b7c40d29f988d713a001bef751a13e336f22d3eab2568e70da843bebd09a7ed6
X-Amz-Date: 20230215T215317Z
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key: MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
Accept-Encoding: gzip

2023/02/15 16:53:17 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:18 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:18 DEBUG : HTTP RESPONSE (req 0xc000258600)
2023/02/15 16:53:18 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Wed, 15 Feb 2023 21:53:19 GMT
Etag: "16dc956e05962b84ad9cd74a05e86797"
Server: AmazonS3
X-Amz-Id-2: d0t9ni34CaO8aeCbD2EBsewtPxEoZImU6jiNh7EN3qCK0vA/xmliXIGgH7udpMCHO0nsBX6Sy7g=
X-Amz-Request-Id: 1BMYZ62QSCM20MX3
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==

2023/02/15 16:53:18 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:18 DEBUG : file.ext: multipart upload starting chunk 4 size 10Mi offset 30Mi/100Mi
2023/02/15 16:53:18 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:18 DEBUG : HTTP REQUEST (req 0xc00061d500)
2023/02/15 16:53:18 DEBUG : PUT /ssec.and.etag/file.ext?partNumber=4&uploadId=qQeIrK7W.SsqPZz4L_htKuZDIntI0zyGiQkthh7hE_VbbtULdwClSq5rgWllCiFjP2v1gr_K.Cb8GW57M3WdbQ-- HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6731.63bf57580.s3-multipart-etag
Content-Length: 10485760
Authorization: XXXX
Content-Md5: sYvsC7H9w0nb031sAnoU/w==
Expect: 100-Continue
X-Amz-Content-Sha256: ecd15cb75c36850aafc572a5671c4d63a125b4a98050077a6f7ab0020cd2c781
X-Amz-Date: 20230215T215318Z
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key: MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
Accept-Encoding: gzip

2023/02/15 16:53:18 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:18 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:18 DEBUG : HTTP RESPONSE (req 0xc00061d500)
2023/02/15 16:53:18 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Wed, 15 Feb 2023 21:53:20 GMT
Etag: "64be66992a5110c4b1151a8249258a1a"
Server: AmazonS3
X-Amz-Id-2: NuKqbYl83TvezfxXkDBu6yB8bIqb1LLbXSCPTSGivq5GVlc0GlZq0/FokpK1XTU7Uyf+msi289o=
X-Amz-Request-Id: E43X092JWSFBJZGM
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==

2023/02/15 16:53:18 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:18 DEBUG : file.ext: multipart upload starting chunk 5 size 10Mi offset 40Mi/100Mi
2023/02/15 16:53:19 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:19 DEBUG : HTTP REQUEST (req 0xc000259100)
2023/02/15 16:53:19 DEBUG : PUT /ssec.and.etag/file.ext?partNumber=5&uploadId=qQeIrK7W.SsqPZz4L_htKuZDIntI0zyGiQkthh7hE_VbbtULdwClSq5rgWllCiFjP2v1gr_K.Cb8GW57M3WdbQ-- HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6731.63bf57580.s3-multipart-etag
Content-Length: 10485760
Authorization: XXXX
Content-Md5: gJYxrJEzqfVXLOsvFz6jQQ==
Expect: 100-Continue
X-Amz-Content-Sha256: a26c65737fcca1dfc233c81acdf08ec1adae431b5d4a15224e0c969a83944df8
X-Amz-Date: 20230215T215319Z
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key: MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
Accept-Encoding: gzip

2023/02/15 16:53:19 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:19 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:19 DEBUG : HTTP RESPONSE (req 0xc000259100)
2023/02/15 16:53:19 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Wed, 15 Feb 2023 21:53:20 GMT
Etag: "4926df0200fe24499524176d6a85e347"
Server: AmazonS3
X-Amz-Id-2: ZpPpvptE/6SLfO+mREp66HCioSRP1Imj4lnnctH6c8hpOBNrWHrryQl2e0mYr0tBjIPuWP4GgQk=
X-Amz-Request-Id: E43MW15YRAHENP8Y
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==

2023/02/15 16:53:19 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:19 DEBUG : file.ext: multipart upload starting chunk 6 size 10Mi offset 50Mi/100Mi
2023/02/15 16:53:19 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:19 DEBUG : HTTP REQUEST (req 0xc000258600)
2023/02/15 16:53:19 DEBUG : PUT /ssec.and.etag/file.ext?partNumber=6&uploadId=qQeIrK7W.SsqPZz4L_htKuZDIntI0zyGiQkthh7hE_VbbtULdwClSq5rgWllCiFjP2v1gr_K.Cb8GW57M3WdbQ-- HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6731.63bf57580.s3-multipart-etag
Content-Length: 10485760
Authorization: XXXX
Content-Md5: aJGM2ZWZCSgu33FnxbvV7w==
Expect: 100-Continue
X-Amz-Content-Sha256: 6d0124c2c572bb5357875365caf03c38def77177473c90b4af245c2709238ede
X-Amz-Date: 20230215T215319Z
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key: MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
Accept-Encoding: gzip

2023/02/15 16:53:19 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:19 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:19 DEBUG : HTTP RESPONSE (req 0xc000258600)
2023/02/15 16:53:19 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Wed, 15 Feb 2023 21:53:21 GMT
Etag: "2b6655c3506481eb1fae6b2e2e7c4b8b"
Server: AmazonS3
X-Amz-Id-2: T6XFNd2oy3J3mHi6XZgmZJD3HH5I/OhrmxWM9vXsZ0efB10b2uuqNb73pLLEuzeHB05ybrqRI/w=
X-Amz-Request-Id: 8QGFW1JMM7QDWDB2
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==

2023/02/15 16:53:19 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:19 DEBUG : file.ext: multipart upload starting chunk 7 size 10Mi offset 60Mi/100Mi
2023/02/15 16:53:20 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:20 DEBUG : HTTP REQUEST (req 0xc00061d500)
2023/02/15 16:53:20 DEBUG : PUT /ssec.and.etag/file.ext?partNumber=7&uploadId=qQeIrK7W.SsqPZz4L_htKuZDIntI0zyGiQkthh7hE_VbbtULdwClSq5rgWllCiFjP2v1gr_K.Cb8GW57M3WdbQ-- HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6731.63bf57580.s3-multipart-etag
Content-Length: 10485760
Authorization: XXXX
Content-Md5: 5maKVeTgIf2hT3wL+PDYHA==
Expect: 100-Continue
X-Amz-Content-Sha256: 21b8c5a170439bf896cd1dd971d474acaa2174ff0e83fcb8bc0a6148694a3c49
X-Amz-Date: 20230215T215319Z
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key: MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
Accept-Encoding: gzip

2023/02/15 16:53:20 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:20 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:20 DEBUG : HTTP RESPONSE (req 0xc00061d500)
2023/02/15 16:53:20 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Wed, 15 Feb 2023 21:53:21 GMT
Etag: "a02e9dbd49039eaf4d6de1fddc5e1a30"
Server: AmazonS3
X-Amz-Id-2: Uh4IA7V+MZCIXfhXB1B3PIRsu4e7fLlwptPR0wCfxLuf6BwysGsD3ka4YzDc7m4hWBBilZGVBNc=
X-Amz-Request-Id: 8QGFSQF6QP9V2XKZ
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==

2023/02/15 16:53:20 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:20 DEBUG : file.ext: multipart upload starting chunk 8 size 10Mi offset 70Mi/100Mi
2023/02/15 16:53:20 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:20 DEBUG : HTTP REQUEST (req 0xc00061d200)
2023/02/15 16:53:20 DEBUG : PUT /ssec.and.etag/file.ext?partNumber=8&uploadId=qQeIrK7W.SsqPZz4L_htKuZDIntI0zyGiQkthh7hE_VbbtULdwClSq5rgWllCiFjP2v1gr_K.Cb8GW57M3WdbQ-- HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6731.63bf57580.s3-multipart-etag
Content-Length: 10485760
Authorization: XXXX
Content-Md5: nXa/4/tr1D18IT4LrD1h0w==
Expect: 100-Continue
X-Amz-Content-Sha256: 54d704181a714821057b380208dd7c9efc76c7089b4a7878b6d5d0c57634ddd4
X-Amz-Date: 20230215T215320Z
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key: MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
Accept-Encoding: gzip

2023/02/15 16:53:20 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:20 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:20 DEBUG : HTTP RESPONSE (req 0xc00061d200)
2023/02/15 16:53:20 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Wed, 15 Feb 2023 21:53:22 GMT
Etag: "afb7bc1f6e0c1f23671cb7116f3b0c63"
Server: AmazonS3
X-Amz-Id-2: Pri1CQ/BXk526Njm4X2T/ycVPxtLp3P9zBz+Uvh61hU79H2Chopetn75RG2haCW1X2S2DVMepGo=
X-Amz-Request-Id: M97YEKRFZ14DHF7V
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==

2023/02/15 16:53:20 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:20 DEBUG : file.ext: multipart upload starting chunk 9 size 10Mi offset 80Mi/100Mi
2023/02/15 16:53:20 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:20 DEBUG : HTTP REQUEST (req 0xc000259100)
2023/02/15 16:53:20 DEBUG : PUT /ssec.and.etag/file.ext?partNumber=9&uploadId=qQeIrK7W.SsqPZz4L_htKuZDIntI0zyGiQkthh7hE_VbbtULdwClSq5rgWllCiFjP2v1gr_K.Cb8GW57M3WdbQ-- HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6731.63bf57580.s3-multipart-etag
Content-Length: 10485760
Authorization: XXXX
Content-Md5: GM7JBdZqFEAWylpWQOPGlQ==
Expect: 100-Continue
X-Amz-Content-Sha256: 1bde37d39b42384bc82be293db6d09b725135806b9bc2c826fad1abfc8f6ea83
X-Amz-Date: 20230215T215320Z
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key: MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
Accept-Encoding: gzip

2023/02/15 16:53:20 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:21 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:21 DEBUG : HTTP RESPONSE (req 0xc000259100)
2023/02/15 16:53:21 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Wed, 15 Feb 2023 21:53:22 GMT
Etag: "dddf3a1ab192f26bb483a3e2778bab13"
Server: AmazonS3
X-Amz-Id-2: mr2tmaKI5hLI4h875UINxrhTuDt6iVf0Vi1vCsTI/0ITHPkeDVV2ZHGtKy43C7XuuQ0MokhzXj0=
X-Amz-Request-Id: M97YQ0AC8ER1M1MH
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==

2023/02/15 16:53:21 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:21 DEBUG : file.ext: multipart upload starting chunk 10 size 10Mi offset 90Mi/100Mi
2023/02/15 16:53:21 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:21 DEBUG : HTTP REQUEST (req 0xc000258600)
2023/02/15 16:53:21 DEBUG : PUT /ssec.and.etag/file.ext?partNumber=10&uploadId=qQeIrK7W.SsqPZz4L_htKuZDIntI0zyGiQkthh7hE_VbbtULdwClSq5rgWllCiFjP2v1gr_K.Cb8GW57M3WdbQ-- HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6731.63bf57580.s3-multipart-etag
Content-Length: 10485760
Authorization: XXXX
Content-Md5: cwgRUwdMsBDBloatp/FKlg==
Expect: 100-Continue
X-Amz-Content-Sha256: 49e0df63597eb31551cecd01646639ec402fa55e9b1d90f569612ead3e5c09f2
X-Amz-Date: 20230215T215321Z
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key: MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
Accept-Encoding: gzip

2023/02/15 16:53:21 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:21 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:21 DEBUG : HTTP RESPONSE (req 0xc000258600)
2023/02/15 16:53:21 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Wed, 15 Feb 2023 21:53:23 GMT
Etag: "adb8b2b761640418856853f3810ac45a"
Server: AmazonS3
X-Amz-Id-2: DlvY+H3zn4KOrjlIssj4ZcUhOZFbdUqoPf5LOerN3s7LmmOyzeX00bMFRRpyzpNDjUgePfektoA=
X-Amz-Request-Id: CDE31TCYB5KTM105
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==

2023/02/15 16:53:21 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:21 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:21 DEBUG : HTTP REQUEST (req 0xc000259a00)
2023/02/15 16:53:21 DEBUG : POST /ssec.and.etag/file.ext?uploadId=qQeIrK7W.SsqPZz4L_htKuZDIntI0zyGiQkthh7hE_VbbtULdwClSq5rgWllCiFjP2v1gr_K.Cb8GW57M3WdbQ-- HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6731.63bf57580.s3-multipart-etag
Content-Length: 1040
Authorization: XXXX
X-Amz-Content-Sha256: 131dfaf473c0f407285990e2ad9c4bdad5efedf303a0e4506e5f02c5d15293da
X-Amz-Date: 20230215T215321Z
Accept-Encoding: gzip

2023/02/15 16:53:21 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:21 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:21 DEBUG : HTTP RESPONSE (req 0xc000259a00)
2023/02/15 16:53:21 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Wed, 15 Feb 2023 21:53:23 GMT
Server: AmazonS3
X-Amz-Id-2: FbV40tX6yyX257mG1Y5fyVQkmO6tTw1zw3BqSXVaySnc74thO/mu6hYEvxqR4BWh4S/gqGs94Po=
X-Amz-Request-Id: CDE3MTJ47JTAPX9N
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256

152
<?xml version="1.0" encoding="UTF-8"?>

<CompleteMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Location>https://s3.us-east-1.amazonaws.com/ssec.and.etag/file.ext</Location><Bucket>ssec.and.etag</Bucket><Key>file.ext</Key><ETag>&quot;c68db040f8a36c164259bcca40c36410-10&quot;</ETag></CompleteMultipartUploadResult>
0

2023/02/15 16:53:21 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:21 DEBUG : file.ext: md5 = 17f18da77e88950c339b903ccc90baea OK
2023/02/15 16:53:21 INFO  : file.ext: Copied (new)
2023/02/15 16:53:21 INFO  : 
Transferred:   	      100 MiB / 100 MiB, 100%, 19.993 MiB/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:         5.6s

2023/02/15 16:53:21 DEBUG : 7 go routines active
9

I disabled the check for sse and etags in the code for you to have a go with

v1.62.0-beta.6741.f793faae4.fix-s3-sse-multipart on branch fix-s3-sse-multipart (uploaded in 15-30 mins)

This might have broken the single part upload checking for SSE-C - so only test with multpart files (set --s3-upload-cutoff 0)

10

Did you have a chance to try the above binary @asdffdsa ?

11

thanks, sorry for the delay,

i was waiting to hear back from wasabi,

Thank you for the report on the issue - it is appreciated! I'll be sure to keep you posted on any updates for this on our end, 
although this would likely be a low-priority bug to work on

i did the testing with AWS

seems that the rclone did not calculate the correct ETag.
Etag differ: expecting 5fa6d124612290af0c85759cb50ce07a-10 but got 0b844664e3e49247b7ed8ae0871ae399-10

on the other hand, my bash script, as before, was able to calculate the ETag.

note: the trick is to use --s3-upload-concurrency=1, then the individual Etags arrive in correct sequence.
when using --s3-upload-concurrency greater than 1, the individual Etags can arrive out of sequence.

etag parts
-------------------------------
cb7accdee5d3207e9411f25c7956f092  -
de3418dbe826686b6962ff91e00dfc0a  -
a22fa1c8c88c232d3d019802a8bcfb61  -
80f92f603c76b3e149d9c01a83f3c5a0  -
c24c827c73e69d2432111f75740d59d3  -
b8e84128e190eec22ce3c1308bc158ee  -
220bb561aa9730f16f4b8074bd9eb549  -
e615b5263a49cde19c6f9868fb5c3441  -
46e790723154d9b45821f6a1b6b7c9c2  -
b8d1cc8eda9c0104d4575c6dd774aec6  -
-------------------------------

etag_from_aws   = 0b844664e3e49247b7ed8ae0871ae399-10
etag_calculated = 0b844664e3e49247b7ed8ae0871ae399-10
The ETags match

and here is the debug log

2023/03/08 09:36:51 DEBUG : rclone: Version "v1.62.0-beta.6741.f793faae4.fix-s3-sse-multipart" starting with parameters ["./rclone" "copy" "./source" "aws_ssec.and.etag:" "--s3-upload-concurrency=1" "--s3-no-head" "--s3-upload-cutoff=0" "--s3-chunk-size=10M" "-vv" "--retries=1" "--files-from=missing-on-dst.txt" "--dump=headers,responses" "--log-file=debug_aws_ssec.txt"]
2023/03/08 09:36:51 DEBUG : Creating backend with remote "./source"
2023/03/08 09:36:51 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2023/03/08 09:36:51 DEBUG : fs cache: renaming cache item "./source" to be canonical "/root/rclone/scripts/minimal/source"
2023/03/08 09:36:51 DEBUG : Creating backend with remote "aws_ssec.and.etag:"
2023/03/08 09:36:51 DEBUG : Creating backend with remote "aws_ssec.and.etag_remote:ssec.and.etag"
2023/03/08 09:36:51 DEBUG : aws_ssec.and.etag_remote: detected overridden config - adding "{bsCrg}" suffix to name
2023/03/08 09:36:51 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2023/03/08 09:36:51 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2023/03/08 09:36:51 DEBUG : fs cache: renaming cache item "aws_ssec.and.etag_remote:ssec.and.etag" to be canonical "aws_ssec.and.etag_remote{bsCrg}:ssec.and.etag"
2023/03/08 09:36:51 DEBUG : fs cache: renaming cache item "aws_ssec.and.etag:" to be canonical "aws_ssec.and.etag_remote{bsCrg}:ssec.and.etag"
2023/03/08 09:36:51 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:51 DEBUG : HTTP REQUEST (req 0xc000a7a800)
2023/03/08 09:36:51 DEBUG : GET /ssec.and.etag?delimiter=%2F&encoding-type=url&list-type=2&max-keys=1000&prefix= HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6741.f793faae4.fix-s3-sse-multipart
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20230308T143651Z
Accept-Encoding: gzip

2023/03/08 09:36:51 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:51 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:36:51 DEBUG : HTTP RESPONSE (req 0xc000a7a800)
2023/03/08 09:36:51 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Wed, 08 Mar 2023 14:36:53 GMT
Server: AmazonS3
X-Amz-Bucket-Region: us-east-1
X-Amz-Id-2: 4k//8l02ShmhKopSNNiVYU71u2EWzDpzo8ujKbtshqmVFH9RCalTy2PtFn7DmFD9lQf6vqm0KdQ=
X-Amz-Request-Id: ST09NFTJC51M5BQR

12c
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>ssec.and.etag</Name><Prefix></Prefix><KeyCount>0</KeyCount><MaxKeys>1000</MaxKeys><Delimiter>/</Delimiter><EncodingType>url</EncodingType><IsTruncated>false</IsTruncated></ListBucketResult>
0

2023/03/08 09:36:51 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:36:51 DEBUG : S3 bucket ssec.and.etag: Waiting for checks to finish
2023/03/08 09:36:51 DEBUG : S3 bucket ssec.and.etag: Waiting for transfers to finish
2023/03/08 09:36:52 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:52 DEBUG : HTTP REQUEST (req 0xc00084c500)
2023/03/08 09:36:52 DEBUG : POST /ssec.and.etag/file.ext?uploads= HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6741.f793faae4.fix-s3-sse-multipart
Content-Length: 0
Authorization: XXXX
Content-Type: application/octet-stream
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20230308T143652Z
X-Amz-Meta-Md5chksum: 4NV7wWg7w2fq8IHaowJf3w==
X-Amz-Meta-Mtime: 1678286111.4957453
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key: MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
X-Amz-Storage-Class: STANDARD
Accept-Encoding: gzip

2023/03/08 09:36:52 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:52 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:36:52 DEBUG : HTTP RESPONSE (req 0xc00084c500)
2023/03/08 09:36:52 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Date: Wed, 08 Mar 2023 14:36:54 GMT
Server: AmazonS3
X-Amz-Id-2: q1VU+k1iI7kDjaYXedqeh4wK6s6f8hR8eYVNKxaRxzyGJLYInpK7xrkuZlhHG8AT9nVvAcB+H0Q=
X-Amz-Request-Id: H6CHC8AAY18V76S2
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==

134
<?xml version="1.0" encoding="UTF-8"?>
<InitiateMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Bucket>ssec.and.etag</Bucket><Key>file.ext</Key><UploadId>FDcCY2scQU_RorCofQ4Hyoh3xND0B7ad387Sao49.BDyPQfK5syqUMPpam.nR.BlfnOjmvs.kq4kOV8BpwMYwg--</UploadId></InitiateMultipartUploadResult>
0

2023/03/08 09:36:52 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:36:52 DEBUG : file.ext: multipart upload starting chunk 1 size 10Mi offset 0/100Mi
2023/03/08 09:36:52 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:52 DEBUG : HTTP REQUEST (req 0xc00084c900)
2023/03/08 09:36:52 DEBUG : PUT /ssec.and.etag/file.ext?partNumber=1&uploadId=FDcCY2scQU_RorCofQ4Hyoh3xND0B7ad387Sao49.BDyPQfK5syqUMPpam.nR.BlfnOjmvs.kq4kOV8BpwMYwg-- HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6741.f793faae4.fix-s3-sse-multipart
Content-Length: 10485760
Authorization: XXXX
Content-Md5: Z3h/lNXi14ku6xzT5Wq2Hw==
Expect: 100-Continue
X-Amz-Content-Sha256: 34fa91a511f950ea0c2a4e0b0fe483f4f472b1ac2314ef419148719956d9d487
X-Amz-Date: 20230308T143652Z
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key: MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
Accept-Encoding: gzip

2023/03/08 09:36:52 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:53 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:36:53 DEBUG : HTTP RESPONSE (req 0xc00084c900)
2023/03/08 09:36:53 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Wed, 08 Mar 2023 14:36:54 GMT
Etag: "cb7accdee5d3207e9411f25c7956f092"
Server: AmazonS3
X-Amz-Id-2: u2iN0+KEe7gREqwpwhFX8jZBDkEKwUnenBuLPnLnQ2Q96F18ENfDI7RAsZCLp/bCuafH9GswBro=
X-Amz-Request-Id: H6CGVS611J4NNZRT
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==

2023/03/08 09:36:53 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:36:53 DEBUG : file.ext: multipart upload starting chunk 2 size 10Mi offset 10Mi/100Mi
2023/03/08 09:36:53 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:53 DEBUG : HTTP REQUEST (req 0xc00084c400)
2023/03/08 09:36:53 DEBUG : PUT /ssec.and.etag/file.ext?partNumber=2&uploadId=FDcCY2scQU_RorCofQ4Hyoh3xND0B7ad387Sao49.BDyPQfK5syqUMPpam.nR.BlfnOjmvs.kq4kOV8BpwMYwg-- HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6741.f793faae4.fix-s3-sse-multipart
Content-Length: 10485760
Authorization: XXXX
Content-Md5: +BBIKFxd1xMMw09diQAXug==
Expect: 100-Continue
X-Amz-Content-Sha256: e181a7bf70f327ebeda3886021879d2c56822db69845c4bbdf267e1a410196ad
X-Amz-Date: 20230308T143653Z
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key: MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
Accept-Encoding: gzip

2023/03/08 09:36:53 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:53 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:36:53 DEBUG : HTTP RESPONSE (req 0xc00084c400)
2023/03/08 09:36:53 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Wed, 08 Mar 2023 14:36:55 GMT
Etag: "de3418dbe826686b6962ff91e00dfc0a"
Server: AmazonS3
X-Amz-Id-2: jPHj0+tgwz/PnyxPN+RLwvP8JqTFYUXutyyCXmx6lFr7BrCOvwO4+1XTfBvCzVLtSt5rJP6ukn0=
X-Amz-Request-Id: HTBF7FMCRKPMBVZN
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==

2023/03/08 09:36:53 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:36:53 DEBUG : file.ext: multipart upload starting chunk 3 size 10Mi offset 20Mi/100Mi
2023/03/08 09:36:53 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:53 DEBUG : HTTP REQUEST (req 0xc000a7a100)
2023/03/08 09:36:53 DEBUG : PUT /ssec.and.etag/file.ext?partNumber=3&uploadId=FDcCY2scQU_RorCofQ4Hyoh3xND0B7ad387Sao49.BDyPQfK5syqUMPpam.nR.BlfnOjmvs.kq4kOV8BpwMYwg-- HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6741.f793faae4.fix-s3-sse-multipart
Content-Length: 10485760
Authorization: XXXX
Content-Md5: jSDg6nOjQCqlg004CtnszA==
Expect: 100-Continue
X-Amz-Content-Sha256: c7b401be36ba98f434fdfda220a719a561bc762c122fa93f34b04867c7d45cb0
X-Amz-Date: 20230308T143653Z
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key: MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
Accept-Encoding: gzip

2023/03/08 09:36:53 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:54 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:36:54 DEBUG : HTTP RESPONSE (req 0xc000a7a100)
2023/03/08 09:36:54 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Wed, 08 Mar 2023 14:36:55 GMT
Etag: "a22fa1c8c88c232d3d019802a8bcfb61"
Server: AmazonS3
X-Amz-Id-2: hUIYZJ90vRfnjaTswMvpghuiXkxClDxKNWZJTDOlC9o3kNl08IYft+zkX7xSADZGj1fmi9C41ao=
X-Amz-Request-Id: HTB5BJ9EHQW5NY63
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==

2023/03/08 09:36:54 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:36:54 DEBUG : file.ext: multipart upload starting chunk 4 size 10Mi offset 30Mi/100Mi
2023/03/08 09:36:54 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:54 DEBUG : HTTP REQUEST (req 0xc00084c400)
2023/03/08 09:36:54 DEBUG : PUT /ssec.and.etag/file.ext?partNumber=4&uploadId=FDcCY2scQU_RorCofQ4Hyoh3xND0B7ad387Sao49.BDyPQfK5syqUMPpam.nR.BlfnOjmvs.kq4kOV8BpwMYwg-- HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6741.f793faae4.fix-s3-sse-multipart
Content-Length: 10485760
Authorization: XXXX
Content-Md5: 8BQVb4Z9X/xItJ7mM9RSog==
Expect: 100-Continue
X-Amz-Content-Sha256: d1b701e89dd402e58e10b2b0572121a1f01e0da304725b3662bdf312dc529add
X-Amz-Date: 20230308T143654Z
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key: MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
Accept-Encoding: gzip

2023/03/08 09:36:54 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:55 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:36:55 DEBUG : HTTP RESPONSE (req 0xc00084c400)
2023/03/08 09:36:55 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Wed, 08 Mar 2023 14:36:56 GMT
Etag: "80f92f603c76b3e149d9c01a83f3c5a0"
Server: AmazonS3
X-Amz-Id-2: ckWh3jJ8DgeAJPYXyj6RT3bvXGlZcZh8Kw7vsqehdpNZJr6JWfo8eAj3QpQ0SU7flZ72sn1NNEE=
X-Amz-Request-Id: M1TE9KD660ZDZ82M
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==

2023/03/08 09:36:55 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:36:55 DEBUG : file.ext: multipart upload starting chunk 5 size 10Mi offset 40Mi/100Mi
2023/03/08 09:36:55 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:55 DEBUG : HTTP REQUEST (req 0xc00084c800)
2023/03/08 09:36:55 DEBUG : PUT /ssec.and.etag/file.ext?partNumber=5&uploadId=FDcCY2scQU_RorCofQ4Hyoh3xND0B7ad387Sao49.BDyPQfK5syqUMPpam.nR.BlfnOjmvs.kq4kOV8BpwMYwg-- HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6741.f793faae4.fix-s3-sse-multipart
Content-Length: 10485760
Authorization: XXXX
Content-Md5: +6WJre7P2U5ftVd0b0qQrg==
Expect: 100-Continue
X-Amz-Content-Sha256: e99abe78681edcd89aea2e0e967152ffe4609e06b3528d751cc814daf874932e
X-Amz-Date: 20230308T143655Z
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key: MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
Accept-Encoding: gzip

2023/03/08 09:36:55 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:55 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:36:55 DEBUG : HTTP RESPONSE (req 0xc00084c800)
2023/03/08 09:36:55 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Wed, 08 Mar 2023 14:36:57 GMT
Etag: "c24c827c73e69d2432111f75740d59d3"
Server: AmazonS3
X-Amz-Id-2: vbTYXgDT6I6mLsoVoLUQ5oIeY+rNM/+qqkFB2lspYozcO7X4pqP3Tu9nHEMlw3k9h8iAH5REsKs=
X-Amz-Request-Id: 34WGC1GP0410VMHR
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==

2023/03/08 09:36:55 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:36:55 DEBUG : file.ext: multipart upload starting chunk 6 size 10Mi offset 50Mi/100Mi
2023/03/08 09:36:56 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:56 DEBUG : HTTP REQUEST (req 0xc00084c400)
2023/03/08 09:36:56 DEBUG : PUT /ssec.and.etag/file.ext?partNumber=6&uploadId=FDcCY2scQU_RorCofQ4Hyoh3xND0B7ad387Sao49.BDyPQfK5syqUMPpam.nR.BlfnOjmvs.kq4kOV8BpwMYwg-- HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6741.f793faae4.fix-s3-sse-multipart
Content-Length: 10485760
Authorization: XXXX
Content-Md5: sVeZXCWnR0VxNqJwBGb+Yw==
Expect: 100-Continue
X-Amz-Content-Sha256: 1aee3c6b41000a2004496c4578785ea57748a301294e212f87467d303b40e827
X-Amz-Date: 20230308T143656Z
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key: MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
Accept-Encoding: gzip

2023/03/08 09:36:56 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:56 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:36:56 DEBUG : HTTP RESPONSE (req 0xc00084c400)
2023/03/08 09:36:56 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Wed, 08 Mar 2023 14:36:58 GMT
Etag: "b8e84128e190eec22ce3c1308bc158ee"
Server: AmazonS3
X-Amz-Id-2: h/2zjZZRMG0a4XYBfC61fjgCGT/8bErS24eNzDHMTHx3v+VduWP2sLZMS39IlS0LWBOGRXDzMBI=
X-Amz-Request-Id: KM844S5SP26VYVD5
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==

2023/03/08 09:36:56 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:36:56 DEBUG : file.ext: multipart upload starting chunk 7 size 10Mi offset 60Mi/100Mi
2023/03/08 09:36:56 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:56 DEBUG : HTTP REQUEST (req 0xc000a7a100)
2023/03/08 09:36:56 DEBUG : PUT /ssec.and.etag/file.ext?partNumber=7&uploadId=FDcCY2scQU_RorCofQ4Hyoh3xND0B7ad387Sao49.BDyPQfK5syqUMPpam.nR.BlfnOjmvs.kq4kOV8BpwMYwg-- HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6741.f793faae4.fix-s3-sse-multipart
Content-Length: 10485760
Authorization: XXXX
Content-Md5: 8anVkoNciJKmmZfdYrWoQQ==
Expect: 100-Continue
X-Amz-Content-Sha256: 238fae0cd097181555bb5109c910647b06af860073072052f509a5828404b68b
X-Amz-Date: 20230308T143656Z
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key: MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
Accept-Encoding: gzip

2023/03/08 09:36:56 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:57 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:36:57 DEBUG : HTTP RESPONSE (req 0xc000a7a100)
2023/03/08 09:36:57 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Wed, 08 Mar 2023 14:36:58 GMT
Etag: "220bb561aa9730f16f4b8074bd9eb549"
Server: AmazonS3
X-Amz-Id-2: ZXhmQ781+dg/cTl0tFzO8+9PEBHrAXuiSkZXycyEFXcQDwiRnE/bbhfeQhWWR0MTHi6v4uGlty8=
X-Amz-Request-Id: KM83PT9P95P0JY4K
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==

2023/03/08 09:36:57 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:36:57 DEBUG : file.ext: multipart upload starting chunk 8 size 10Mi offset 70Mi/100Mi
2023/03/08 09:36:57 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:57 DEBUG : HTTP REQUEST (req 0xc000a7a700)
2023/03/08 09:36:57 DEBUG : PUT /ssec.and.etag/file.ext?partNumber=8&uploadId=FDcCY2scQU_RorCofQ4Hyoh3xND0B7ad387Sao49.BDyPQfK5syqUMPpam.nR.BlfnOjmvs.kq4kOV8BpwMYwg-- HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6741.f793faae4.fix-s3-sse-multipart
Content-Length: 10485760
Authorization: XXXX
Content-Md5: wSrOoZjwyJMAe4Pxfd7dFg==
Expect: 100-Continue
X-Amz-Content-Sha256: 03bdeb2a0abd5fb5587812eb0d90b863682218713f5c10ee1b530824bb9f0256
X-Amz-Date: 20230308T143657Z
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key: MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
Accept-Encoding: gzip

2023/03/08 09:36:57 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:58 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:36:58 DEBUG : HTTP RESPONSE (req 0xc000a7a700)
2023/03/08 09:36:58 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Wed, 08 Mar 2023 14:36:59 GMT
Etag: "e615b5263a49cde19c6f9868fb5c3441"
Server: AmazonS3
X-Amz-Id-2: zq2X9fUBh2T2N+cFjCcMC6Kp5CPMi91ZXhzfhFDZo2W6OvGCmsNt9Mvsz40JRC5AMWPKdCtSfzc=
X-Amz-Request-Id: QMGTPQ8HV8JWDG6Z
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==

2023/03/08 09:36:58 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:36:58 DEBUG : file.ext: multipart upload starting chunk 9 size 10Mi offset 80Mi/100Mi
2023/03/08 09:36:58 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:58 DEBUG : HTTP REQUEST (req 0xc0008acc00)
2023/03/08 09:36:58 DEBUG : PUT /ssec.and.etag/file.ext?partNumber=9&uploadId=FDcCY2scQU_RorCofQ4Hyoh3xND0B7ad387Sao49.BDyPQfK5syqUMPpam.nR.BlfnOjmvs.kq4kOV8BpwMYwg-- HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6741.f793faae4.fix-s3-sse-multipart
Content-Length: 10485760
Authorization: XXXX
Content-Md5: e8eTA22VCBMBJIOKJXAKWw==
Expect: 100-Continue
X-Amz-Content-Sha256: 9dbdce8c67deea24a10b88b9ab1f2d4929feb7ea49c283d33e5ac936625877ca
X-Amz-Date: 20230308T143658Z
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key: MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
Accept-Encoding: gzip

2023/03/08 09:36:58 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:59 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:36:59 DEBUG : HTTP RESPONSE (req 0xc0008acc00)
2023/03/08 09:36:59 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Wed, 08 Mar 2023 14:37:00 GMT
Etag: "46e790723154d9b45821f6a1b6b7c9c2"
Server: AmazonS3
X-Amz-Id-2: eXGmJUKqEtRUq4tGu3kDJlyNhgnOud2pmGcD2msCnO9g/4QEHyT4iInkkyeLmXP8MRSB0Ll72Uk=
X-Amz-Request-Id: 1WXNNQXM34HRPRPG
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==

2023/03/08 09:36:59 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:36:59 DEBUG : file.ext: multipart upload starting chunk 10 size 10Mi offset 90Mi/100Mi
2023/03/08 09:36:59 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:59 DEBUG : HTTP REQUEST (req 0xc000a7a000)
2023/03/08 09:36:59 DEBUG : PUT /ssec.and.etag/file.ext?partNumber=10&uploadId=FDcCY2scQU_RorCofQ4Hyoh3xND0B7ad387Sao49.BDyPQfK5syqUMPpam.nR.BlfnOjmvs.kq4kOV8BpwMYwg-- HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6741.f793faae4.fix-s3-sse-multipart
Content-Length: 10485760
Authorization: XXXX
Content-Md5: hpWEdc4fUwNiTObOA/xFSQ==
Expect: 100-Continue
X-Amz-Content-Sha256: a279cf7e76603504fbf16eee7103191d8fdc3d5c36c576b4123e0ad6fb40ea62
X-Amz-Date: 20230308T143659Z
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key: MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
Accept-Encoding: gzip

2023/03/08 09:36:59 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:59 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:36:59 DEBUG : HTTP RESPONSE (req 0xc000a7a000)
2023/03/08 09:36:59 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Wed, 08 Mar 2023 14:37:01 GMT
Etag: "b8d1cc8eda9c0104d4575c6dd774aec6"
Server: AmazonS3
X-Amz-Id-2: XwQiH3STPy7YxtFRCN04on6N7LgZss5mofIhh0rP+AxzS0nlpNR/1EtDl2nxEqYrgJBIucPgi2c=
X-Amz-Request-Id: AQ5Y5XRFWK4YS658
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==

2023/03/08 09:36:59 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:36:59 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:59 DEBUG : HTTP REQUEST (req 0xc0008acc00)
2023/03/08 09:36:59 DEBUG : POST /ssec.and.etag/file.ext?uploadId=FDcCY2scQU_RorCofQ4Hyoh3xND0B7ad387Sao49.BDyPQfK5syqUMPpam.nR.BlfnOjmvs.kq4kOV8BpwMYwg-- HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6741.f793faae4.fix-s3-sse-multipart
Content-Length: 1040
Authorization: XXXX
X-Amz-Content-Sha256: 7a683b9981004bacdcc50a339a6458c8cc54e172cc0f12666ecd72b47774dfcd
X-Amz-Date: 20230308T143659Z
Accept-Encoding: gzip

2023/03/08 09:36:59 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/03/08 09:36:59 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:36:59 DEBUG : HTTP RESPONSE (req 0xc0008acc00)
2023/03/08 09:37:00 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Wed, 08 Mar 2023 14:37:01 GMT
Server: AmazonS3
X-Amz-Id-2: +E38CKpH7XAbllZQ9jUMt/mDaA/Hiamz4XwroY7Qx3keLuQIFGSrXzhdZArhlAjQ/M+To8J9mJo=
X-Amz-Request-Id: AQ5VXBMFCFSC6KBM
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256

152
<?xml version="1.0" encoding="UTF-8"?>

<CompleteMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Location>https://s3.us-east-1.amazonaws.com/ssec.and.etag/file.ext</Location><Bucket>ssec.and.etag</Bucket><Key>file.ext</Key><ETag>&quot;0b844664e3e49247b7ed8ae0871ae399-10&quot;</ETag></CompleteMultipartUploadResult>
0

2023/03/08 09:37:00 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/03/08 09:37:00 ERROR : file.ext: Failed to copy: multipart upload corrupted: Etag differ: expecting 5fa6d124612290af0c85759cb50ce07a-10 but got 0b844664e3e49247b7ed8ae0871ae399-10
2023/03/08 09:37:00 ERROR : Attempt 1/1 failed with 1 errors and: multipart upload corrupted: Etag differ: expecting 5fa6d124612290af0c85759cb50ce07a-10 but got 0b844664e3e49247b7ed8ae0871ae399-10
2023/03/08 09:37:00 INFO  : 
Transferred:   	      100 MiB / 100 MiB, 100%, 12.499 MiB/s, ETA 0s
Errors:                 1 (retrying may help)
Elapsed time:         8.5s

2023/03/08 09:37:00 DEBUG : 8 go routines active
2023/03/08 09:37:00 Failed to copy: multipart upload corrupted: Etag differ: expecting 5fa6d124612290af0c85759cb50ce07a-10 but got 0b844664e3e49247b7ed8ae0871ae399-10
12

Rclone is calculating the ETags in order, but it is using the md5sum of the input, not the ETag returned after uploading a single part. I mis-understood what you were doing - when you say ETag you mean the ETag returned from the upload a single part.

As you said it is a little more complicated to calculate the overall ETag from the ETags returned taking into account concurrency but rclone is doign this already

Try this - this uses the returned ETags from the upload rather than the md5sum rclone calculates and see if it works.

I think this is potentially less reliable that what rclone is doing at the moment but it will be interesting to see if it works for you.

v1.62.0-beta.6784.b4555d98f.fix-s3-sse-multipart on branch fix-s3-sse-multipart (uploaded in 15-30 mins)

13

the beta returned the correct final Etag value :white_check_mark:

if rclone simply collects all the individual Etags returned from aws, and from that, rclone computes the final Etag,
then i agree, that is not a solution.

my goal:

  1. as rclone uploads each part, rclone calculates the Etag for that part.
  2. after the part is uploaded, aws returns the Etag of that part.
  3. rclone compares its calculated Etag against the aws Etag.
  4. is there is a mismatch, then rclone would raise an error.
  5. after the upload completes, compare the rclone calculated final Etag against the aws final Etag.

the other use for the final Etag is when copying between two s3 providers, let's say wasabi->aws
if the Etag returned by wasabi matches the Etag returned by aws, that would be a valid transfer.
and in this case, does not rely on rclone calculations to verify transfer.

14

I think the problem with this scenario is that rclone can't calculate the ETag if SSE is in use in step 1. as the ETag is the MD5 sum of the encrypted data and rclone has no access to that.

If SSE is not in use then this is what rclone does already (more or less).

So if SSE is in use rclone could be doing

1, upload a part
2. after the part is uploaded collect the ETag
3. after the upload completes compare the rclone calculated final ETag against the aws final ETag.

This is what the latest binary I sent you does.

So what I was thinking is that it should have a different mode if SSE is enabled.

This is all getting rather complicated though!

1 Like
15

yes, it is, let's leave this as it is. thanks very much

1 Like
16

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.