in my testing, with multipart+ssec, rclone can check.
if the below looks valid, perhaps we can have rclone check the etag,
perhaps --s3-use-multipart-etag=ssec
and then i can do more testing against other s3 providers, such as wasabi
i wrote a script that
- from the rclone debug log, extract all the individual ETag, in the example below, 10 parts so 10 ETag values.
- calculate the overall ETag from all individual ETag.
- compare the calculated ETag against the overall ETag returned from aws
here is the output
etag parts
-------------------------------
1330e1275b556ab6702bca9438f62c15 -
ae55d3ddf52e33d45140a5be6dacb925 -
16dc956e05962b84ad9cd74a05e86797 -
64be66992a5110c4b1151a8249258a1a -
4926df0200fe24499524176d6a85e347 -
2b6655c3506481eb1fae6b2e2e7c4b8b -
a02e9dbd49039eaf4d6de1fddc5e1a30 -
afb7bc1f6e0c1f23671cb7116f3b0c63 -
dddf3a1ab192f26bb483a3e2778bab13 -
adb8b2b761640418856853f3810ac45a -
-------------------------------
etag_from_aws = c68db040f8a36c164259bcca40c36410-10
etag_calculated = c68db040f8a36c164259bcca40c36410-10
The ETags match
now, here is the script. pass the path of the rclone log file as the first argument.
cat $1 | grep -oP 'Etag: "\K[^"]+' | sed 's/$/ -/' > etags_from_parts.txt
cat $1 | grep -oP '<ETag>"\K[^"]+' > etag_from_aws.txt
etag_from_aws=$(cat etag_from_aws.txt)
temp1=$(xxd -r -p etags_from_parts.txt | md5sum)
temp2=$(echo $temp1 | tail -c+1 | head -c32)
partnum=$(wc -l < etags_from_parts.txt)
etag_calculated="${temp2}-${partnum}"
echo; echo etag parts ;echo "-------------------------------"
cat etags_from_parts.txt
echo "-------------------------------"; echo
echo "etag_from_aws = ${etag_from_aws}"
echo "etag_calculated = ${etag_calculated}"
if [ $etag_from_aws = $etag_calculated ];
then echo 'The ETags match'
else echo 'The ETags do not match'
fi
and here is the debug log,
2023/02/15 16:53:16 DEBUG : rclone: Version "v1.62.0-beta.6731.63bf57580.s3-multipart-etag" starting with parameters ["./rclone" "copy" "./source" "aws_ssec.and.etag:" "--s3-no-head" "--s3-upload-concurrency=1" "--s3-upload-cutoff=0" "--s3-chunk-size=10M" "-vv" "--retries=1" "--files-from=missing-on-dst.txt" "--dump=headers,responses" "--log-file=debug_aws_ssec.txt"]
2023/02/15 16:53:16 DEBUG : Creating backend with remote "./source"
2023/02/15 16:53:16 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2023/02/15 16:53:16 DEBUG : fs cache: renaming cache item "./source" to be canonical "/root/rclone/scripts/minimal/source"
2023/02/15 16:53:16 DEBUG : Creating backend with remote "aws_ssec.and.etag:"
2023/02/15 16:53:16 DEBUG : Creating backend with remote "aws_ssec.and.etag_remote:ssec.and.etag"
2023/02/15 16:53:16 DEBUG : aws_ssec.and.etag_remote: detected overridden config - adding "{bsCrg}" suffix to name
2023/02/15 16:53:16 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2023/02/15 16:53:16 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2023/02/15 16:53:16 DEBUG : fs cache: renaming cache item "aws_ssec.and.etag_remote:ssec.and.etag" to be canonical "aws_ssec.and.etag_remote{bsCrg}:ssec.and.etag"
2023/02/15 16:53:16 DEBUG : fs cache: renaming cache item "aws_ssec.and.etag:" to be canonical "aws_ssec.and.etag_remote{bsCrg}:ssec.and.etag"
2023/02/15 16:53:16 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:16 DEBUG : HTTP REQUEST (req 0xc000259000)
2023/02/15 16:53:16 DEBUG : GET /ssec.and.etag?delimiter=%2F&encoding-type=url&list-type=2&max-keys=1000&prefix= HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6731.63bf57580.s3-multipart-etag
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20230215T215316Z
Accept-Encoding: gzip
2023/02/15 16:53:16 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:16 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:16 DEBUG : HTTP RESPONSE (req 0xc000259000)
2023/02/15 16:53:16 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Wed, 15 Feb 2023 21:53:18 GMT
Server: AmazonS3
X-Amz-Bucket-Region: us-east-1
X-Amz-Id-2: JOVw+xqzartxtWtqo0XCuv6ySfyG5YY6vn6k59kadv6aAGacq7iKKQp1AbXRUSQg95zbK8Aw5qo=
X-Amz-Request-Id: B4SFV1X36Y230X9N
12c
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>ssec.and.etag</Name><Prefix></Prefix><KeyCount>0</KeyCount><MaxKeys>1000</MaxKeys><Delimiter>/</Delimiter><EncodingType>url</EncodingType><IsTruncated>false</IsTruncated></ListBucketResult>
0
2023/02/15 16:53:16 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:16 DEBUG : S3 bucket ssec.and.etag: Waiting for checks to finish
2023/02/15 16:53:16 DEBUG : S3 bucket ssec.and.etag: Waiting for transfers to finish
2023/02/15 16:53:16 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:16 DEBUG : HTTP REQUEST (req 0xc00084b300)
2023/02/15 16:53:16 DEBUG : POST /ssec.and.etag/file.ext?uploads= HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6731.63bf57580.s3-multipart-etag
Content-Length: 0
Authorization: XXXX
Content-Type: application/octet-stream
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20230215T215316Z
X-Amz-Meta-Md5chksum: F/GNp36IlQwzm5A8zJC66g==
X-Amz-Meta-Mtime: 1676496659.931
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key: MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
X-Amz-Storage-Class: STANDARD
Accept-Encoding: gzip
2023/02/15 16:53:16 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:16 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:16 DEBUG : HTTP RESPONSE (req 0xc00084b300)
2023/02/15 16:53:16 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Date: Wed, 15 Feb 2023 21:53:18 GMT
Server: AmazonS3
X-Amz-Id-2: p524Cdfmo7IqJdQ6QiTelJUlRSvNG52xoSLhIfOJCqNZALnopbIMVJZPBCfC1STA08MIzqjtLbY=
X-Amz-Request-Id: B4S1B3KZYDG7S49B
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
134
<?xml version="1.0" encoding="UTF-8"?>
<InitiateMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Bucket>ssec.and.etag</Bucket><Key>file.ext</Key><UploadId>qQeIrK7W.SsqPZz4L_htKuZDIntI0zyGiQkthh7hE_VbbtULdwClSq5rgWllCiFjP2v1gr_K.Cb8GW57M3WdbQ--</UploadId></InitiateMultipartUploadResult>
0
2023/02/15 16:53:16 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:16 DEBUG : file.ext: multipart upload starting chunk 1 size 10Mi offset 0/100Mi
2023/02/15 16:53:16 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:16 DEBUG : HTTP REQUEST (req 0xc000258600)
2023/02/15 16:53:16 DEBUG : PUT /ssec.and.etag/file.ext?partNumber=1&uploadId=qQeIrK7W.SsqPZz4L_htKuZDIntI0zyGiQkthh7hE_VbbtULdwClSq5rgWllCiFjP2v1gr_K.Cb8GW57M3WdbQ-- HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6731.63bf57580.s3-multipart-etag
Content-Length: 10485760
Authorization: XXXX
Content-Md5: lkipYzu2pL322EXK7RTjZg==
Expect: 100-Continue
X-Amz-Content-Sha256: 2466895a4ae323cdda7c41cad749ac3e4032175594e7c39ede3128a213150e96
X-Amz-Date: 20230215T215316Z
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key: MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
Accept-Encoding: gzip
2023/02/15 16:53:16 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:17 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:17 DEBUG : HTTP RESPONSE (req 0xc000258600)
2023/02/15 16:53:17 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Wed, 15 Feb 2023 21:53:18 GMT
Etag: "1330e1275b556ab6702bca9438f62c15"
Server: AmazonS3
X-Amz-Id-2: FisMLP0rbEZDCzc014QH8JZ7zz9ACSqPMeEiNYljBKj+8VcdgqZA7wkszvmEP7I1Aj4uG5TGHzA=
X-Amz-Request-Id: B4SEHY70HQTX435R
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
2023/02/15 16:53:17 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:17 DEBUG : file.ext: multipart upload starting chunk 2 size 10Mi offset 10Mi/100Mi
2023/02/15 16:53:17 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:17 DEBUG : HTTP REQUEST (req 0xc000259300)
2023/02/15 16:53:17 DEBUG : PUT /ssec.and.etag/file.ext?partNumber=2&uploadId=qQeIrK7W.SsqPZz4L_htKuZDIntI0zyGiQkthh7hE_VbbtULdwClSq5rgWllCiFjP2v1gr_K.Cb8GW57M3WdbQ-- HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6731.63bf57580.s3-multipart-etag
Content-Length: 10485760
Authorization: XXXX
Content-Md5: FcJqqZe9uR+rzQBZPw2IwA==
Expect: 100-Continue
X-Amz-Content-Sha256: 2eec1b74ef5a55e40ec1329794275b6197a48175512460e4df9910a982d040e8
X-Amz-Date: 20230215T215317Z
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key: MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
Accept-Encoding: gzip
2023/02/15 16:53:17 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:17 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:17 DEBUG : HTTP RESPONSE (req 0xc000259300)
2023/02/15 16:53:17 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Wed, 15 Feb 2023 21:53:19 GMT
Etag: "ae55d3ddf52e33d45140a5be6dacb925"
Server: AmazonS3
X-Amz-Id-2: chAaKXGwpvLjBcTk1xvQAPSvZqCWbWOhHunEAVUkv2o0kAJcgbZMGj4LXxW6K7xoYaHuP4sbn2o=
X-Amz-Request-Id: 1BMKXWE7N7F9H6ZH
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
2023/02/15 16:53:17 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:17 DEBUG : file.ext: multipart upload starting chunk 3 size 10Mi offset 20Mi/100Mi
2023/02/15 16:53:17 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:17 DEBUG : HTTP REQUEST (req 0xc000258600)
2023/02/15 16:53:17 DEBUG : PUT /ssec.and.etag/file.ext?partNumber=3&uploadId=qQeIrK7W.SsqPZz4L_htKuZDIntI0zyGiQkthh7hE_VbbtULdwClSq5rgWllCiFjP2v1gr_K.Cb8GW57M3WdbQ-- HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6731.63bf57580.s3-multipart-etag
Content-Length: 10485760
Authorization: XXXX
Content-Md5: B0FYoWA53eaRcsAGB6oNfg==
Expect: 100-Continue
X-Amz-Content-Sha256: b7c40d29f988d713a001bef751a13e336f22d3eab2568e70da843bebd09a7ed6
X-Amz-Date: 20230215T215317Z
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key: MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
Accept-Encoding: gzip
2023/02/15 16:53:17 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:18 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:18 DEBUG : HTTP RESPONSE (req 0xc000258600)
2023/02/15 16:53:18 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Wed, 15 Feb 2023 21:53:19 GMT
Etag: "16dc956e05962b84ad9cd74a05e86797"
Server: AmazonS3
X-Amz-Id-2: d0t9ni34CaO8aeCbD2EBsewtPxEoZImU6jiNh7EN3qCK0vA/xmliXIGgH7udpMCHO0nsBX6Sy7g=
X-Amz-Request-Id: 1BMYZ62QSCM20MX3
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
2023/02/15 16:53:18 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:18 DEBUG : file.ext: multipart upload starting chunk 4 size 10Mi offset 30Mi/100Mi
2023/02/15 16:53:18 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:18 DEBUG : HTTP REQUEST (req 0xc00061d500)
2023/02/15 16:53:18 DEBUG : PUT /ssec.and.etag/file.ext?partNumber=4&uploadId=qQeIrK7W.SsqPZz4L_htKuZDIntI0zyGiQkthh7hE_VbbtULdwClSq5rgWllCiFjP2v1gr_K.Cb8GW57M3WdbQ-- HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6731.63bf57580.s3-multipart-etag
Content-Length: 10485760
Authorization: XXXX
Content-Md5: sYvsC7H9w0nb031sAnoU/w==
Expect: 100-Continue
X-Amz-Content-Sha256: ecd15cb75c36850aafc572a5671c4d63a125b4a98050077a6f7ab0020cd2c781
X-Amz-Date: 20230215T215318Z
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key: MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
Accept-Encoding: gzip
2023/02/15 16:53:18 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:18 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:18 DEBUG : HTTP RESPONSE (req 0xc00061d500)
2023/02/15 16:53:18 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Wed, 15 Feb 2023 21:53:20 GMT
Etag: "64be66992a5110c4b1151a8249258a1a"
Server: AmazonS3
X-Amz-Id-2: NuKqbYl83TvezfxXkDBu6yB8bIqb1LLbXSCPTSGivq5GVlc0GlZq0/FokpK1XTU7Uyf+msi289o=
X-Amz-Request-Id: E43X092JWSFBJZGM
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
2023/02/15 16:53:18 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:18 DEBUG : file.ext: multipart upload starting chunk 5 size 10Mi offset 40Mi/100Mi
2023/02/15 16:53:19 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:19 DEBUG : HTTP REQUEST (req 0xc000259100)
2023/02/15 16:53:19 DEBUG : PUT /ssec.and.etag/file.ext?partNumber=5&uploadId=qQeIrK7W.SsqPZz4L_htKuZDIntI0zyGiQkthh7hE_VbbtULdwClSq5rgWllCiFjP2v1gr_K.Cb8GW57M3WdbQ-- HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6731.63bf57580.s3-multipart-etag
Content-Length: 10485760
Authorization: XXXX
Content-Md5: gJYxrJEzqfVXLOsvFz6jQQ==
Expect: 100-Continue
X-Amz-Content-Sha256: a26c65737fcca1dfc233c81acdf08ec1adae431b5d4a15224e0c969a83944df8
X-Amz-Date: 20230215T215319Z
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key: MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
Accept-Encoding: gzip
2023/02/15 16:53:19 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:19 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:19 DEBUG : HTTP RESPONSE (req 0xc000259100)
2023/02/15 16:53:19 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Wed, 15 Feb 2023 21:53:20 GMT
Etag: "4926df0200fe24499524176d6a85e347"
Server: AmazonS3
X-Amz-Id-2: ZpPpvptE/6SLfO+mREp66HCioSRP1Imj4lnnctH6c8hpOBNrWHrryQl2e0mYr0tBjIPuWP4GgQk=
X-Amz-Request-Id: E43MW15YRAHENP8Y
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
2023/02/15 16:53:19 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:19 DEBUG : file.ext: multipart upload starting chunk 6 size 10Mi offset 50Mi/100Mi
2023/02/15 16:53:19 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:19 DEBUG : HTTP REQUEST (req 0xc000258600)
2023/02/15 16:53:19 DEBUG : PUT /ssec.and.etag/file.ext?partNumber=6&uploadId=qQeIrK7W.SsqPZz4L_htKuZDIntI0zyGiQkthh7hE_VbbtULdwClSq5rgWllCiFjP2v1gr_K.Cb8GW57M3WdbQ-- HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6731.63bf57580.s3-multipart-etag
Content-Length: 10485760
Authorization: XXXX
Content-Md5: aJGM2ZWZCSgu33FnxbvV7w==
Expect: 100-Continue
X-Amz-Content-Sha256: 6d0124c2c572bb5357875365caf03c38def77177473c90b4af245c2709238ede
X-Amz-Date: 20230215T215319Z
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key: MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
Accept-Encoding: gzip
2023/02/15 16:53:19 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:19 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:19 DEBUG : HTTP RESPONSE (req 0xc000258600)
2023/02/15 16:53:19 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Wed, 15 Feb 2023 21:53:21 GMT
Etag: "2b6655c3506481eb1fae6b2e2e7c4b8b"
Server: AmazonS3
X-Amz-Id-2: T6XFNd2oy3J3mHi6XZgmZJD3HH5I/OhrmxWM9vXsZ0efB10b2uuqNb73pLLEuzeHB05ybrqRI/w=
X-Amz-Request-Id: 8QGFW1JMM7QDWDB2
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
2023/02/15 16:53:19 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:19 DEBUG : file.ext: multipart upload starting chunk 7 size 10Mi offset 60Mi/100Mi
2023/02/15 16:53:20 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:20 DEBUG : HTTP REQUEST (req 0xc00061d500)
2023/02/15 16:53:20 DEBUG : PUT /ssec.and.etag/file.ext?partNumber=7&uploadId=qQeIrK7W.SsqPZz4L_htKuZDIntI0zyGiQkthh7hE_VbbtULdwClSq5rgWllCiFjP2v1gr_K.Cb8GW57M3WdbQ-- HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6731.63bf57580.s3-multipart-etag
Content-Length: 10485760
Authorization: XXXX
Content-Md5: 5maKVeTgIf2hT3wL+PDYHA==
Expect: 100-Continue
X-Amz-Content-Sha256: 21b8c5a170439bf896cd1dd971d474acaa2174ff0e83fcb8bc0a6148694a3c49
X-Amz-Date: 20230215T215319Z
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key: MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
Accept-Encoding: gzip
2023/02/15 16:53:20 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:20 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:20 DEBUG : HTTP RESPONSE (req 0xc00061d500)
2023/02/15 16:53:20 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Wed, 15 Feb 2023 21:53:21 GMT
Etag: "a02e9dbd49039eaf4d6de1fddc5e1a30"
Server: AmazonS3
X-Amz-Id-2: Uh4IA7V+MZCIXfhXB1B3PIRsu4e7fLlwptPR0wCfxLuf6BwysGsD3ka4YzDc7m4hWBBilZGVBNc=
X-Amz-Request-Id: 8QGFSQF6QP9V2XKZ
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
2023/02/15 16:53:20 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:20 DEBUG : file.ext: multipart upload starting chunk 8 size 10Mi offset 70Mi/100Mi
2023/02/15 16:53:20 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:20 DEBUG : HTTP REQUEST (req 0xc00061d200)
2023/02/15 16:53:20 DEBUG : PUT /ssec.and.etag/file.ext?partNumber=8&uploadId=qQeIrK7W.SsqPZz4L_htKuZDIntI0zyGiQkthh7hE_VbbtULdwClSq5rgWllCiFjP2v1gr_K.Cb8GW57M3WdbQ-- HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6731.63bf57580.s3-multipart-etag
Content-Length: 10485760
Authorization: XXXX
Content-Md5: nXa/4/tr1D18IT4LrD1h0w==
Expect: 100-Continue
X-Amz-Content-Sha256: 54d704181a714821057b380208dd7c9efc76c7089b4a7878b6d5d0c57634ddd4
X-Amz-Date: 20230215T215320Z
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key: MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
Accept-Encoding: gzip
2023/02/15 16:53:20 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:20 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:20 DEBUG : HTTP RESPONSE (req 0xc00061d200)
2023/02/15 16:53:20 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Wed, 15 Feb 2023 21:53:22 GMT
Etag: "afb7bc1f6e0c1f23671cb7116f3b0c63"
Server: AmazonS3
X-Amz-Id-2: Pri1CQ/BXk526Njm4X2T/ycVPxtLp3P9zBz+Uvh61hU79H2Chopetn75RG2haCW1X2S2DVMepGo=
X-Amz-Request-Id: M97YEKRFZ14DHF7V
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
2023/02/15 16:53:20 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:20 DEBUG : file.ext: multipart upload starting chunk 9 size 10Mi offset 80Mi/100Mi
2023/02/15 16:53:20 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:20 DEBUG : HTTP REQUEST (req 0xc000259100)
2023/02/15 16:53:20 DEBUG : PUT /ssec.and.etag/file.ext?partNumber=9&uploadId=qQeIrK7W.SsqPZz4L_htKuZDIntI0zyGiQkthh7hE_VbbtULdwClSq5rgWllCiFjP2v1gr_K.Cb8GW57M3WdbQ-- HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6731.63bf57580.s3-multipart-etag
Content-Length: 10485760
Authorization: XXXX
Content-Md5: GM7JBdZqFEAWylpWQOPGlQ==
Expect: 100-Continue
X-Amz-Content-Sha256: 1bde37d39b42384bc82be293db6d09b725135806b9bc2c826fad1abfc8f6ea83
X-Amz-Date: 20230215T215320Z
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key: MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
Accept-Encoding: gzip
2023/02/15 16:53:20 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:21 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:21 DEBUG : HTTP RESPONSE (req 0xc000259100)
2023/02/15 16:53:21 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Wed, 15 Feb 2023 21:53:22 GMT
Etag: "dddf3a1ab192f26bb483a3e2778bab13"
Server: AmazonS3
X-Amz-Id-2: mr2tmaKI5hLI4h875UINxrhTuDt6iVf0Vi1vCsTI/0ITHPkeDVV2ZHGtKy43C7XuuQ0MokhzXj0=
X-Amz-Request-Id: M97YQ0AC8ER1M1MH
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
2023/02/15 16:53:21 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:21 DEBUG : file.ext: multipart upload starting chunk 10 size 10Mi offset 90Mi/100Mi
2023/02/15 16:53:21 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:21 DEBUG : HTTP REQUEST (req 0xc000258600)
2023/02/15 16:53:21 DEBUG : PUT /ssec.and.etag/file.ext?partNumber=10&uploadId=qQeIrK7W.SsqPZz4L_htKuZDIntI0zyGiQkthh7hE_VbbtULdwClSq5rgWllCiFjP2v1gr_K.Cb8GW57M3WdbQ-- HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6731.63bf57580.s3-multipart-etag
Content-Length: 10485760
Authorization: XXXX
Content-Md5: cwgRUwdMsBDBloatp/FKlg==
Expect: 100-Continue
X-Amz-Content-Sha256: 49e0df63597eb31551cecd01646639ec402fa55e9b1d90f569612ead3e5c09f2
X-Amz-Date: 20230215T215321Z
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key: MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
Accept-Encoding: gzip
2023/02/15 16:53:21 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:21 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:21 DEBUG : HTTP RESPONSE (req 0xc000258600)
2023/02/15 16:53:21 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Wed, 15 Feb 2023 21:53:23 GMT
Etag: "adb8b2b761640418856853f3810ac45a"
Server: AmazonS3
X-Amz-Id-2: DlvY+H3zn4KOrjlIssj4ZcUhOZFbdUqoPf5LOerN3s7LmmOyzeX00bMFRRpyzpNDjUgePfektoA=
X-Amz-Request-Id: CDE31TCYB5KTM105
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key-Md5: dnF5x6K/8ZZRzpfSlMMM+w==
2023/02/15 16:53:21 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:21 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:21 DEBUG : HTTP REQUEST (req 0xc000259a00)
2023/02/15 16:53:21 DEBUG : POST /ssec.and.etag/file.ext?uploadId=qQeIrK7W.SsqPZz4L_htKuZDIntI0zyGiQkthh7hE_VbbtULdwClSq5rgWllCiFjP2v1gr_K.Cb8GW57M3WdbQ-- HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.62.0-beta.6731.63bf57580.s3-multipart-etag
Content-Length: 1040
Authorization: XXXX
X-Amz-Content-Sha256: 131dfaf473c0f407285990e2ad9c4bdad5efedf303a0e4506e5f02c5d15293da
X-Amz-Date: 20230215T215321Z
Accept-Encoding: gzip
2023/02/15 16:53:21 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/15 16:53:21 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:21 DEBUG : HTTP RESPONSE (req 0xc000259a00)
2023/02/15 16:53:21 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Wed, 15 Feb 2023 21:53:23 GMT
Server: AmazonS3
X-Amz-Id-2: FbV40tX6yyX257mG1Y5fyVQkmO6tTw1zw3BqSXVaySnc74thO/mu6hYEvxqR4BWh4S/gqGs94Po=
X-Amz-Request-Id: CDE3MTJ47JTAPX9N
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
152
<?xml version="1.0" encoding="UTF-8"?>
<CompleteMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Location>https://s3.us-east-1.amazonaws.com/ssec.and.etag/file.ext</Location><Bucket>ssec.and.etag</Bucket><Key>file.ext</Key><ETag>"c68db040f8a36c164259bcca40c36410-10"</ETag></CompleteMultipartUploadResult>
0
2023/02/15 16:53:21 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/15 16:53:21 DEBUG : file.ext: md5 = 17f18da77e88950c339b903ccc90baea OK
2023/02/15 16:53:21 INFO : file.ext: Copied (new)
2023/02/15 16:53:21 INFO :
Transferred: 100 MiB / 100 MiB, 100%, 19.993 MiB/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 5.6s
2023/02/15 16:53:21 DEBUG : 7 go routines active