What is the problem you are having with rclone?
after file transfer completes, force rclone to compare rclone calculated ETag against provider calculated ETag.
from the log, looks like wasabi returns the Etag, yet rclone does not use it?
<CompleteMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Location>https://azork.s3.us-east-2.wasabisys.com/file.ext</Location><Bucket>azork</Bucket><Key>file.ext</Key>
<ETag>"195c549125ec2fe75e9f3e8444b4a98c-2"</ETag></CompleteMultipartUploadResult>
i want this
DEBUG : file.ext: Multipart upload Etag: 195c549125ec2fe75e9f3e8444b4a98c-2 OK
DEBUG : file.ext: md5 = 7246aff55b4ba5cdd90912d8819d5a0f OK
INFO : file.ext: Copied (new)
instead i get this
DEBUG : file.ext: md5 = 7246aff55b4ba5cdd90912d8819d5a0f OK
INFO : file.ext: Copied (new)
note: the issue occurs when using this locked down bucket policy and as a result, have to use --s3-no-head
{ "Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": "arn:aws:iam::redacted:user/zork"},
"Action": ["s3:ListBucket", "s3:PutObject"],
"Resource": ["arn:aws:s3:::azork/*", "arn:aws:s3:::azork"]}]
}
Run the command 'rclone version' and share the full output of the command.
rclone v1.61.1
- os/version: ubuntu 22.04 (64 bit)
- os/kernel: 5.15.79.1-microsoft-standard-WSL2 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.19.4
- go/linking: static
- go/tags: none
Which cloud storage system are you using? (eg Google Drive)
wasabi, s3 provider
The command you were trying to run (eg rclone copy /tmp remote:tmp)
rclone check ./source azork: --size-only --one-way --missing-on-dst=missing-on-dst.txt -vv
rclone copy ./source azork: --s3-use-multipart-etag=true --s3-no-head --s3-upload-cutoff=0 --s3-chunk-size=19M --files-from=missing-on-dst.txt --dump=headers,responses
The rclone config contents with secrets removed.
[wasabi_azork_remote]
type = s3
provider = Wasabi
access_key_id = redacted
secret_access_key = redacted
endpoint = s3.us-east-2.wasabisys.com
[azork]
type = alias
remote = wasabi_azork_remote:azork
A log from the command with the -vv flag
rclone copy ./source azork: --files-from=missing-on-dst.txt --s3-no-head --s3-upload-cutoff=0 --s3-chunk-size=19M --dump=headers,responses -vv
2023/02/12 10:51:55 DEBUG : rclone: Version "v1.61.1" starting with parameters ["rclone" "copy" "./source" "azork:" "-vv" "--retries=1" "--files-from=missing-on-dst.txt" "--s3-no-head" "--s3-upload-cutoff=0" "--s3-chunk-size=19M" "--dump=headers,responses" "--s3-use-multipart-etag=true"]
2023/02/12 10:51:55 DEBUG : Creating backend with remote "./source"
2023/02/12 10:51:55 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2023/02/12 10:51:55 DEBUG : fs cache: renaming cache item "./source" to be canonical "/root/rclone/scripts/source"
2023/02/12 10:51:55 DEBUG : Creating backend with remote "azork:"
2023/02/12 10:51:55 DEBUG : Creating backend with remote "wasabi_azork_remote:azork"
2023/02/12 10:51:55 DEBUG : wasabi_azork_remote: detected overridden config - adding "{EWbYf}" suffix to name
2023/02/12 10:51:55 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2023/02/12 10:51:55 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2023/02/12 10:51:55 DEBUG : fs cache: renaming cache item "wasabi_azork_remote:azork" to be canonical "wasabi_azork_remote{EWbYf}:azork"
2023/02/12 10:51:55 DEBUG : fs cache: renaming cache item "azork:" to be canonical "wasabi_azork_remote{EWbYf}:azork"
2023/02/12 10:51:55 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/12 10:51:55 DEBUG : HTTP REQUEST (req 0xc0008dab00)
2023/02/12 10:51:55 DEBUG : GET /?delimiter=%2F&encoding-type=url&list-type=2&max-keys=1000&prefix= HTTP/1.1
Host: azork.s3.us-east-2.wasabisys.com
User-Agent: rclone/v1.61.1
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20230212T155155Z
Accept-Encoding: gzip
2023/02/12 10:51:55 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/12 10:51:56 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/12 10:51:56 DEBUG : HTTP RESPONSE (req 0xc0008dab00)
2023/02/12 10:51:56 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Sun, 12 Feb 2023 15:51:56 GMT
Server: WasabiS3/7.10.1198-2022-12-14-39a7a2e69e (B33-U25)
X-Amz-Bucket-Region: us-east-2
X-Amz-Id-2: X4qgZ7hl17jxppDU8TwveloepcI5HOS/35d52ERkD+ZaapD2I54WoLiD0xrrmUknSlnKo+nXhFrm
X-Amz-Request-Id: 5FB66F630CB45F6D:A
124
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>azork</Name><Prefix></Prefix><KeyCount>2</KeyCount><MaxKeys>1000</MaxKeys><Delimiter>/</Delimiter><EncodingType>url</EncodingType><IsTruncated>false</IsTruncated></ListBucketResult>
0
2023/02/12 10:51:56 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/12 10:51:56 DEBUG : S3 bucket azork: Waiting for checks to finish
2023/02/12 10:51:56 DEBUG : S3 bucket azork: Waiting for transfers to finish
2023/02/12 10:51:56 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/12 10:51:56 DEBUG : HTTP REQUEST (req 0xc0008dac00)
2023/02/12 10:51:56 DEBUG : POST /file.ext?uploads= HTTP/1.1
Host: azork.s3.us-east-2.wasabisys.com
User-Agent: rclone/v1.61.1
Content-Length: 0
Authorization: XXXX
Content-Type: application/octet-stream
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20230212T155156Z
X-Amz-Meta-Md5chksum: ckav9VtLpc3ZCRLYgZ1aDw==
X-Amz-Meta-Mtime: 1676217114.643056688
Accept-Encoding: gzip
2023/02/12 10:51:56 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/12 10:51:56 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/12 10:51:56 DEBUG : HTTP RESPONSE (req 0xc0008dac00)
2023/02/12 10:51:56 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Date: Sun, 12 Feb 2023 15:51:56 GMT
Server: WasabiS3/7.10.1198-2022-12-14-39a7a2e69e (B33-U25)
X-Amz-Id-2: L6JhR7W0Ia+5VO8AifCM7SELoTQk5dzmoMrqgTgqu247vlGGFTq5D74c25c/TogRGe0xUZKcSQdn
X-Amz-Request-Id: 16C09B78C9A0714D:A
154
<?xml version="1.0" encoding="UTF-8"?>
<InitiateMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Bucket>azork</Bucket><Key>file.ext</Key><UploadId>u3_2_FKr98usheWHYacDfnD1gHfeIeltoJhNX18KYYbQXXwsy9cE_e8UTVLNB1wtXcnIRuvUTQM5v8Cib2g8nNByc9xs2mfU6GFUqaYxHoG0DBDYkSTDJCBCF9bulvGb</UploadId></InitiateMultipartUploadResult>
0
2023/02/12 10:51:56 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/12 10:51:56 DEBUG : file.ext: multipart upload starting chunk 1 size 19Mi offset 0/20Mi
2023/02/12 10:51:56 DEBUG : file.ext: multipart upload starting chunk 2 size 1Mi offset 19Mi/20Mi
2023/02/12 10:51:56 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/12 10:51:56 DEBUG : HTTP REQUEST (req 0xc0008dab00)
2023/02/12 10:51:56 DEBUG : PUT /file.ext?partNumber=2&uploadId=u3_2_FKr98usheWHYacDfnD1gHfeIeltoJhNX18KYYbQXXwsy9cE_e8UTVLNB1wtXcnIRuvUTQM5v8Cib2g8nNByc9xs2mfU6GFUqaYxHoG0DBDYkSTDJCBCF9bulvGb HTTP/1.1
Host: azork.s3.us-east-2.wasabisys.com
User-Agent: rclone/v1.61.1
Content-Length: 1048576
Authorization: XXXX
Content-Md5: aUHYpoSioPsWAqy4QOT46Q==
X-Amz-Content-Sha256: 66de6f9b5138899a8c8decc8f1cf1cbb99fa8e2c52b2a0e6b45eeeae2dffbf1e
X-Amz-Date: 20230212T155156Z
Accept-Encoding: gzip
2023/02/12 10:51:56 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/12 10:51:56 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/12 10:51:56 DEBUG : HTTP REQUEST (req 0xc0006f0200)
2023/02/12 10:51:56 DEBUG : PUT /file.ext?partNumber=1&uploadId=u3_2_FKr98usheWHYacDfnD1gHfeIeltoJhNX18KYYbQXXwsy9cE_e8UTVLNB1wtXcnIRuvUTQM5v8Cib2g8nNByc9xs2mfU6GFUqaYxHoG0DBDYkSTDJCBCF9bulvGb HTTP/1.1
Host: azork.s3.us-east-2.wasabisys.com
User-Agent: rclone/v1.61.1
Content-Length: 19922944
Authorization: XXXX
Content-Md5: n/jrKfsFqxQ2DIIWZVnx+g==
Expect: 100-Continue
X-Amz-Content-Sha256: 0f033eb7f8431b96208e604338ed327d2c5f398908b6d21cd3e6fb6e4009c363
X-Amz-Date: 20230212T155156Z
Accept-Encoding: gzip
2023/02/12 10:51:56 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/12 10:51:56 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/12 10:51:56 DEBUG : HTTP RESPONSE (req 0xc0008dab00)
2023/02/12 10:51:56 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Sun, 12 Feb 2023 15:51:56 GMT
Etag: "6941d8a684a2a0fb1602acb840e4f8e9"
Server: WasabiS3/7.10.1198-2022-12-14-39a7a2e69e (B33-U25)
X-Amz-Id-2: e6wV/ViO9CtVj2n7hMyGYcgE21T9h15AhRV4r3AfHLWTSvElZENqqJkXNNgnANVqG2l7y3KLqMul
X-Amz-Request-Id: 2B942E69FCD0A8CD:A
2023/02/12 10:51:56 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/12 10:51:57 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/12 10:51:57 DEBUG : HTTP RESPONSE (req 0xc0006f0200)
2023/02/12 10:51:57 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Sun, 12 Feb 2023 15:51:57 GMT
Etag: "9ff8eb29fb05ab14360c82166559f1fa"
Server: WasabiS3/7.10.1198-2022-12-14-39a7a2e69e (XB27-U40)
X-Amz-Id-2: lxaRY6gsBItfbColH33BMqFQJQBLgBQy2YTkg2FNXW9j71GAh6udosnblCatyaI6/RaT7/4tZwxV
X-Amz-Request-Id: 7C40CBE5BA852FE4:A
2023/02/12 10:51:57 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/12 10:51:57 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/12 10:51:57 DEBUG : HTTP REQUEST (req 0xc0008dad00)
2023/02/12 10:51:57 DEBUG : POST /file.ext?uploadId=u3_2_FKr98usheWHYacDfnD1gHfeIeltoJhNX18KYYbQXXwsy9cE_e8UTVLNB1wtXcnIRuvUTQM5v8Cib2g8nNByc9xs2mfU6GFUqaYxHoG0DBDYkSTDJCBCF9bulvGb HTTP/1.1
Host: azork.s3.us-east-2.wasabisys.com
User-Agent: rclone/v1.61.1
Content-Length: 287
Authorization: XXXX
X-Amz-Content-Sha256: d204a9abf0a8b8d18bd291b0675bdfd86c490a4520ec7c40a1dd66d6cdc297e7
X-Amz-Date: 20230212T155157Z
Accept-Encoding: gzip
2023/02/12 10:51:57 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/12 10:51:57 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/12 10:51:57 DEBUG : HTTP RESPONSE (req 0xc0008dad00)
2023/02/12 10:51:57 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Sun, 12 Feb 2023 15:51:57 GMT
Server: WasabiS3/7.10.1198-2022-12-14-39a7a2e69e (XB27-U40)
X-Amz-Id-2: 00AA7cnTDV/PHuAAMYxow7mqzA9WbrveA6qN5Yc14L/a3eKC+GAl/gZY8Rr4DumgfA2xjzggQYIM
X-Amz-Request-Id: 622C041D60C48EF7:A
X-Amz-Version-Id: 001676217116418210501-ugBQes-Rwr
140
<?xml version="1.0" encoding="UTF-8"?>
<CompleteMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Location>https://azork.s3.us-east-2.wasabisys.com/file.ext</Location><Bucket>azork</Bucket><Key>file.ext</Key><ETag>"195c549125ec2fe75e9f3e8444b4a98c-2"</ETag></CompleteMultipartUploadResult>
0
2023/02/12 10:51:57 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/12 10:51:57 DEBUG : file.ext: md5 = 7246aff55b4ba5cdd90912d8819d5a0f OK
2023/02/12 10:51:57 INFO : file.ext: Copied (new)
2023/02/12 10:51:57 INFO :
Transferred: 20 MiB / 20 MiB, 100%, 19.978 MiB/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 1.5s
2023/02/12 10:51:57 DEBUG : 9 go routines active
