What is the problem you are having with rclone?
no problem, asking question.
took a look at the source code and i wanted to confirm
for a multi-part upload file copy, from s3 bucket to s3 bucket.
how does rclone ensure that the dest file matches the source?
does rclone compare the dest ETag to the source ETag?
or is rclone performing a running calculation and using that?
Run the command 'rclone version' and share the full output of the command.
rclone v1.57.0
- os/version: Microsoft Windows 10 Pro 2009 (64 bit)
- os/kernel: 10.0.22000.493 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.17.2
- go/linking: dynamic
- go/tags: cmount
Which cloud storage system are you using? (eg Google Drive)
for testing, both remotes are wasabi s3, each remote is using a unique bucket in a different endpoint.
for production, a large amount of TB between wasabi to aws s3 deep glacier.
The command you were trying to run (eg rclone copy /tmp remote:tmp
)
rclone copy source:zorksource/file.txt dest:zorkdest --s3-chunk-size=5Mi --s3-upload-concurrency=1 --s3-upload-cutoff=1b --s3-copy-cutoff=1b --s3-no-check-bucket --dump=headers --retries=1 --low-level-retries=1 --stats=0 --log-level=DEBUG --log-file=.\\log.source2dest.txt
The rclone config contents with secrets removed.
[source]
type = s3
provider = Wasabi
access_key_id =
secret_access_key =
endpoint = s3.us-east-2.wasabisys.com
[dest]
type = s3
provider = Wasabi
access_key_id =
secret_access_key =
endpoint = s3.us-west-1.wasabisys.com
A log from the command with the -vv
flag
DEBUG : rclone: Version "v1.57.0" starting with parameters ["c:\\data\\rclone\\scripts\\rclone.exe" "copy" "source:zorksource/file.txt" "dest:zorkdest" "--s3-chunk-size=5Mi" "--s3-upload-concurrency=1" "--s3-upload-cutoff=1b" "--s3-copy-cutoff=1b" "--s3-no-check-bucket" "--dump=headers" "--retries=1" "--low-level-retries=1" "--stats=0" "--log-level=DEBUG" "--log-file=.\\\\log.source2dest.txt"]
DEBUG : Creating backend with remote "source:zorksource/file.txt"
DEBUG : Using config file from "c:\\data\\rclone\\scripts\\barantec.conf"
DEBUG : source: detected overridden config - adding "{8c8ZL}" suffix to name
DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : HTTP REQUEST (req 0xc0002aa700)
DEBUG : HEAD /zorksource/file.txt HTTP/1.1
Host: s3.us-east-2.wasabisys.com
User-Agent: rclone/v1.57.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220213T234341Z
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : HTTP RESPONSE (req 0xc0002aa700)
DEBUG : HTTP/1.1 200 OK
Content-Length: 6291456
Accept-Ranges: bytes
Content-Type: text/plain; charset=utf-8
Date: Sun, 13 Feb 2022 23:43:42 GMT
Etag: "41b392acc7c51164aee62c3b96386db0-2"
Last-Modified: Sun, 13 Feb 2022 23:43:42 GMT
Server: WasabiS3/7.2.3941-2022-02-03-4a1b388 (head5)
X-Amz-Id-2: YVF9//Tyzt8BImh7VHD8At3+UXbNJKuQh3ewq9qvMkK3X2wjLxJD0GBLlrVzbG9x9xjgfGu9P9Lp
X-Amz-Meta-Md5chksum: /Ic8BG9v3xOE0suoBuadbA==
X-Amz-Meta-Mtime: 1644783735.4892583
X-Amz-Request-Id: D98F2FDB790D8114
DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : fs cache: adding new entry for parent of "source:zorksource/file.txt", "source{8c8ZL}:zorksource"
DEBUG : Creating backend with remote "dest:zorkdest"
DEBUG : dest: detected overridden config - adding "{8c8ZL}" suffix to name
DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
DEBUG : fs cache: renaming cache item "dest:zorkdest" to be canonical "dest{8c8ZL}:zorkdest"
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : HTTP REQUEST (req 0xc00072aa00)
DEBUG : HEAD /zorksource/file.txt HTTP/1.1
Host: s3.us-east-2.wasabisys.com
User-Agent: rclone/v1.57.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220213T234341Z
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : HTTP RESPONSE (req 0xc00072aa00)
DEBUG : HTTP/1.1 200 OK
Content-Length: 6291456
Accept-Ranges: bytes
Content-Type: text/plain; charset=utf-8
Date: Sun, 13 Feb 2022 23:43:42 GMT
Etag: "41b392acc7c51164aee62c3b96386db0-2"
Last-Modified: Sun, 13 Feb 2022 23:43:42 GMT
Server: WasabiS3/7.2.3941-2022-02-03-4a1b388 (head5)
X-Amz-Id-2: rtNfvt0/C6UBIlNG+RHK/BMZVXkuS2U6uPLtd1zXaSi97kNLXyGur1E57DUzQTg/wxGmFn2APBex
X-Amz-Meta-Md5chksum: /Ic8BG9v3xOE0suoBuadbA==
X-Amz-Meta-Mtime: 1644783735.4892583
X-Amz-Request-Id: DE69F657A35FF116
DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : HTTP REQUEST (req 0xc0008b2500)
DEBUG : HEAD /zorkdest/file.txt HTTP/1.1
Host: s3.us-west-1.wasabisys.com
User-Agent: rclone/v1.57.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220213T234341Z
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : HTTP RESPONSE (req 0xc0008b2500)
DEBUG : HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Sun, 13 Feb 2022 23:43:43 GMT
Server: WasabiS3/7.2.3941-2022-02-03-4a1b388 (head10)
X-Amz-Id-2: dqKwzefEW5cnvAaZDB+MG95LlQEo67fc32/Fx6wx8mKsY7e9kiya8B9hydj11Cuv9s492ll0cO8r
X-Amz-Request-Id: 46522B47289FFAFC
DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : file.txt: Need to transfer - File not found at Destination
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : HTTP REQUEST (req 0xc0002aac00)
DEBUG : GET /zorksource/file.txt HTTP/1.1
Host: s3.us-east-2.wasabisys.com
User-Agent: rclone/v1.57.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220213T234341Z
Accept-Encoding: gzip
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : HTTP RESPONSE (req 0xc0002aac00)
DEBUG : HTTP/1.1 200 OK
Content-Length: 6291456
Accept-Ranges: bytes
Content-Type: text/plain; charset=utf-8
Date: Sun, 13 Feb 2022 23:43:43 GMT
Etag: "41b392acc7c51164aee62c3b96386db0-2"
Last-Modified: Sun, 13 Feb 2022 23:43:42 GMT
Server: WasabiS3/7.2.3941-2022-02-03-4a1b388 (head5)
X-Amz-Id-2: 7BuD7zs5o559Z4McBtJWk2dWRt0Nk0vzPQBEkbb+mYhbF2pLb3fvS8hkNICUwC1v3SvSLrrEKCl3
X-Amz-Meta-Md5chksum: /Ic8BG9v3xOE0suoBuadbA==
X-Amz-Meta-Mtime: 1644783735.4892583
X-Amz-Request-Id: 9238FEBDB60FCACF
DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : HTTP REQUEST (req 0xc000198000)
DEBUG : POST /zorkdest/file.txt?uploads= HTTP/1.1
Host: s3.us-west-1.wasabisys.com
User-Agent: rclone/v1.57.0
Content-Length: 0
Authorization: XXXX
Content-Type: text/plain; charset=utf-8
X-Amz-Acl: private
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220213T234342Z
X-Amz-Meta-Md5chksum: /Ic8BG9v3xOE0suoBuadbA==
X-Amz-Meta-Mtime: 1644783735.4892583
Accept-Encoding: gzip
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : HTTP RESPONSE (req 0xc000198000)
DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Date: Sun, 13 Feb 2022 23:43:43 GMT
Server: WasabiS3/7.2.3941-2022-02-03-4a1b388 (head10)
X-Amz-Id-2: U571W0ZJ0WrPlRRrqAC9QQQlZiQfYoAOl3L/a49Xh3LQ+LN8T3qQwCBxZzkyDHClqNisbHo1mqG7
X-Amz-Request-Id: EB5FE36FCBE8F9E0
DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : file.txt: multipart upload starting chunk 1 size 5Mi offset 0/6Mi
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : HTTP REQUEST (req 0xc00072a100)
DEBUG : PUT /zorkdest/file.txt?partNumber=1&uploadId=AfpDu1N9bmy3tAgoW2YlZHT4bPZgdsgVTPKAsYoalWcufVqKVgNvfOED7sTiEF9VukkAwraXnEA13yoIZAOVc48n7s0agsbqpUBL7KICe7hUIY9iHN3WVD212SiJBVQ1 HTTP/1.1
Host: s3.us-west-1.wasabisys.com
User-Agent: rclone/v1.57.0
Content-Length: 5242880
Authorization: XXXX
Content-Md5: YwohxPIQ30r59ZKUjSA5hQ==
Expect: 100-Continue
X-Amz-Content-Sha256: 0a48bce725f726914568e9497b4b8417585773ce0330cee548c2f4b4f3b14867
X-Amz-Date: 20220213T234342Z
Accept-Encoding: gzip
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : HTTP RESPONSE (req 0xc00072a100)
DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Sun, 13 Feb 2022 23:43:45 GMT
Etag: "630a21c4f210df4af9f592948d203985"
Server: WasabiS3/7.2.3941-2022-02-03-4a1b388 (head10)
X-Amz-Id-2: f6l66O5sM5+uqScb5W6TvSIHSOKyy0zyD24madIf68cFiZe4laIhbjLQ0zPUtr+5g4swGWOVV5LT
X-Amz-Request-Id: 74D1C31BC1203A59
DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : file.txt: multipart upload starting chunk 2 size 1Mi offset 5Mi/6Mi
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : HTTP REQUEST (req 0xc00072a700)
DEBUG : PUT /zorkdest/file.txt?partNumber=2&uploadId=AfpDu1N9bmy3tAgoW2YlZHT4bPZgdsgVTPKAsYoalWcufVqKVgNvfOED7sTiEF9VukkAwraXnEA13yoIZAOVc48n7s0agsbqpUBL7KICe7hUIY9iHN3WVD212SiJBVQ1 HTTP/1.1
Host: s3.us-west-1.wasabisys.com
User-Agent: rclone/v1.57.0
Content-Length: 1048576
Authorization: XXXX
Content-Md5: 0eVmJbdTd7pOjMMMSFQ14g==
X-Amz-Content-Sha256: ef7eae4416a1bd60b9f1c65e0cf68dc3d6d5bb9c21967883402d67f9c81608c7
X-Amz-Date: 20220213T234344Z
Accept-Encoding: gzip
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : HTTP RESPONSE (req 0xc00072a700)
DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Sun, 13 Feb 2022 23:43:46 GMT
Etag: "d1e56625b75377ba4e8cc30c485435e2"
Server: WasabiS3/7.2.3941-2022-02-03-4a1b388 (head10)
X-Amz-Id-2: Mr3gbZ7OHCCGBMVRP9QMpmnFFW8koTQoDADuF8XYQWGjHpWXW48bAKRP/ZhWf40OMGh8e6dW8SuV
X-Amz-Request-Id: 5C30696FDB80155C
DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : HTTP REQUEST (req 0xc0000cba00)
DEBUG : POST /zorkdest/file.txt?uploadId=AfpDu1N9bmy3tAgoW2YlZHT4bPZgdsgVTPKAsYoalWcufVqKVgNvfOED7sTiEF9VukkAwraXnEA13yoIZAOVc48n7s0agsbqpUBL7KICe7hUIY9iHN3WVD212SiJBVQ1 HTTP/1.1
Host: s3.us-west-1.wasabisys.com
User-Agent: rclone/v1.57.0
Content-Length: 287
Authorization: XXXX
X-Amz-Content-Sha256: 80b32595fec8ff5be7c3c8507e10db819d70c6bc538354e2bc9ece3db8de5fc7
X-Amz-Date: 20220213T234344Z
Accept-Encoding: gzip
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : HTTP RESPONSE (req 0xc0000cba00)
DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Sun, 13 Feb 2022 23:43:46 GMT
Server: WasabiS3/7.2.3941-2022-02-03-4a1b388 (head10)
X-Amz-Id-2: pXMXV0lSRvvFqbhHNDvZHvUS0LsUt/MxD/bHjMk0uco2z9yDHNzeQ3lfAtOBMtamtbmqz/P+sy9C
X-Amz-Request-Id: D53D32A2969BC8AB
DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : HTTP REQUEST (req 0xc0008b2400)
DEBUG : HEAD /zorkdest/file.txt HTTP/1.1
Host: s3.us-west-1.wasabisys.com
User-Agent: rclone/v1.57.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220213T234344Z
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : HTTP RESPONSE (req 0xc0008b2400)
DEBUG : HTTP/1.1 200 OK
Content-Length: 6291456
Accept-Ranges: bytes
Content-Type: text/plain; charset=utf-8
Date: Sun, 13 Feb 2022 23:43:46 GMT
Etag: "41b392acc7c51164aee62c3b96386db0-2"
Last-Modified: Sun, 13 Feb 2022 23:43:46 GMT
Server: WasabiS3/7.2.3941-2022-02-03-4a1b388 (head10)
X-Amz-Id-2: W2AH9A8T2I3ptS5TpnbeEJ9bvY3A5zFxLEpkncfFKzOYk9ibyEJsg+iz5Svibc+9ZZ5ojVIUfrPj
X-Amz-Meta-Md5chksum: /Ic8BG9v3xOE0suoBuadbA==
X-Amz-Meta-Mtime: 1644783735.4892583
X-Amz-Request-Id: 6438DD0BEF1A64DC
DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
DEBUG : file.txt: md5 = fc873c046f6fdf1384d2cba806e69d6c OK
INFO : file.txt: Copied (new)
DEBUG : 10 go routines active