Rclone to Linode Object Storage Fails for all but Small Files

Suspected Bug
1

What is the problem you are having with rclone?

Uploads of 10mb or less work correctly, but 200mb or larger fail consistently with NoSuchUpload errors.

Run the command 'rclone version' and share the full output of the command.

rclone v1.64.0

  • os/version: debian 10.7 (64 bit)
  • os/kernel: 4.19.0-12-amd64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.21.1
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Akamai Linode Object Storage (S3 Ceph)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy ap.part-aa linodes3:/AP-prod-web-01/ap.part-aa

The rclone config contents with secrets removed.

[linodes3]
type = s3
provider = Ceph
endpoint = https://<serverinstance>.us-east-1.linodeobjects.com
acl = private

A log from the command with the -vv flag

I'll update when I can, but after downgrading to version 1.61.1, the issue went away. Here is a piece of the error log:

rclone copy /bhg/ap-prod-web-01.tgz.part-iw linodes3:/AP-prod-web-01/ap-prod-web-01.tgz.part-iw
2023/09/20 04:55:59 ERROR : ap-prod-web-01.tgz.part-iw: Failed to copy: failed to upload chunk 1 with 5242880 bytes: NoSuchUpload:
status code: 404, request id: tx000007c634bb4c4734927-00650ab39f-477dc889-default, host id:
2023/09/20 04:55:59 ERROR : Attempt 1/3 failed with 1 errors and: failed to upload chunk 1 with 5242880 bytes: NoSuchUpload:
2

welcome to the forum,

please, copy a single file and post the full output

3

Thanks. It's a file, but it works (when rclone is working).

the result is a folder /AP-prod-web-01/ap-prod-web-01.tgz.part-iw with a file ap-prod-web-01.tgz.part-iw inside it.

We're just setting up Object Storage for the first time, and didn't figure rclone was the issue. We'll straighten out the folders.

4

Okay, here is the command with the -vv output on 1.64.0.
The exact same command with 1.61.1 works.

File size is 1,073,741,824 bytes.

rclone -vv copy /mnt/tmp-ap-backup2/bhg/ap-prod-web-01.tgz.part-in linodes3:/AP-prod-web-01/
2023/09/20 10:32:45 DEBUG : rclone: Version "v1.64.0" starting with parameters ["rclone" "-vv" "copy" "/mnt/tmp-ap-backup2/bhg/ap-prod-web-01.tgz.part-in" "linodes3:/AP-prod-web-01/"]
2023/09/20 10:32:45 DEBUG : Creating backend with remote "/mnt/tmp-ap-backup2/bhg/ap-prod-web-01.tgz.part-in"
2023/09/20 10:32:45 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2023/09/20 10:32:45 DEBUG : fs cache: adding new entry for parent of "/mnt/tmp-ap-backup2/bhg/ap-prod-web-01.tgz.part-in", "/mnt/tmp-ap-backup2/bhg"
2023/09/20 10:32:45 DEBUG : Creating backend with remote "linodes3:/AP-prod-web-01/"
2023/09/20 10:32:45 DEBUG : Resolving service "s3" region "us-east-1"
2023/09/20 10:32:45 DEBUG : fs cache: renaming cache item "linodes3:/AP-prod-web-01/" to be canonical "linodes3:AP-prod-web-01"
2023/09/20 10:32:45 DEBUG : ap-prod-web-01.tgz.part-in: Need to transfer - File not found at Destination
2023/09/20 10:32:45 INFO  : S3 bucket AP-prod-web-01: Bucket "AP-prod-web-01" created with ACL "private"
2023/09/20 10:32:51 DEBUG : ap-prod-web-01.tgz.part-in: open chunk writer: started multipart upload: 2~Hi3Td0Rtr6SqtI1Ho8PHtYP0w5U1FW2
2023/09/20 10:32:51 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 4
2023/09/20 10:32:51 DEBUG : ap-prod-web-01.tgz.part-in: Starting multi-thread copy with 205 chunks of size 5Mi with 4 parallel streams
2023/09/20 10:32:51 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 4/205 (15728640-20971520) size 5Mi starting
2023/09/20 10:32:51 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 2/205 (5242880-10485760) size 5Mi starting
2023/09/20 10:32:51 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 3/205 (10485760-15728640) size 5Mi starting
2023/09/20 10:32:51 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 1/205 (0-5242880) size 5Mi starting
2023/09/20 10:32:52 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 2/205 failed: multi-thread copy: failed to write chunk: failed to upload chunk 2 with 5242880 bytes: NoSuchUpload:
        status code: 404, request id: tx00000815ba055478a27cc-00650b0294-48aacb84-default, host id:
2023/09/20 10:32:52 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 3/205 failed: multi-thread copy: failed to write chunk: failed to upload chunk 3 with 5242880 bytes: RequestCanceled: request context canceled
caused by: context canceled
2023/09/20 10:32:52 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 4/205 failed: multi-thread copy: failed to write chunk: failed to upload chunk 4 with 5242880 bytes: NoSuchUpload:
        status code: 404, request id: tx000006f6790c533066275-00650b0294-487a9a9a-default, host id:
2023/09/20 10:32:52 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 5/205 (20971520-26214400) size 5Mi starting
2023/09/20 10:32:52 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 1/205 failed: multi-thread copy: failed to write chunk: failed to upload chunk 1 with 5242880 bytes: NoSuchUpload:
        status code: 404, request id: tx000008c2c09718d922aa1-00650b0294-484a448f-default, host id:
2023/09/20 10:32:52 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 5/205 failed: multi-thread copy: failed to write chunk: failed to upload chunk 5 with 5242880 bytes: RequestCanceled: request context canceled
caused by: context canceled
2023/09/20 10:32:52 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: cancelling transfer on exit
2023/09/20 10:32:52 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: abort failed: failed to abort multipart upload "2~Hi3Td0Rtr6SqtI1Ho8PHtYP0w5U1FW2": NoSuchUpload:
        status code: 404, request id: tx000006dd9d3d0385c630b-00650b0294-48b0c730-default, host id:
2023/09/20 10:32:52 ERROR : ap-prod-web-01.tgz.part-in: Failed to copy: multi-thread copy: failed to write chunk: failed to upload chunk 2 with 5242880 bytes: NoSuchUpload:
        status code: 404, request id: tx00000815ba055478a27cc-00650b0294-48aacb84-default, host id:
2023/09/20 10:32:52 ERROR : Attempt 1/3 failed with 1 errors and: multi-thread copy: failed to write chunk: failed to upload chunk 2 with 5242880 bytes: NoSuchUpload:
        status code: 404, request id: tx00000815ba055478a27cc-00650b0294-48aacb84-default, host id:
2023/09/20 10:32:52 DEBUG : ap-prod-web-01.tgz.part-in: Need to transfer - File not found at Destination
2023/09/20 10:32:58 DEBUG : ap-prod-web-01.tgz.part-in: open chunk writer: started multipart upload: 2~l77PBr2f0GKMNB85FmER2x1NHvCXjGO
2023/09/20 10:32:58 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 4
2023/09/20 10:32:58 DEBUG : ap-prod-web-01.tgz.part-in: Starting multi-thread copy with 205 chunks of size 5Mi with 4 parallel streams
2023/09/20 10:32:58 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 4/205 (15728640-20971520) size 5Mi starting
2023/09/20 10:32:58 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 1/205 (0-5242880) size 5Mi starting
2023/09/20 10:32:58 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 2/205 (5242880-10485760) size 5Mi starting
2023/09/20 10:32:58 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 3/205 (10485760-15728640) size 5Mi starting
2023/09/20 10:32:58 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 2/205 failed: multi-thread copy: failed to write chunk: failed to upload chunk 2 with 5242880 bytes: NoSuchUpload:
        status code: 404, request id: tx00000a9310d970224ed62-00650b029a-487043e8-default, host id:
2023/09/20 10:32:58 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 5/205 (20971520-26214400) size 5Mi starting
2023/09/20 10:32:58 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 3/205 failed: multi-thread copy: failed to write chunk: failed to upload chunk 3 with 5242880 bytes: RequestCanceled: request context canceled
caused by: context canceled
2023/09/20 10:32:58 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 1/205 failed: multi-thread copy: failed to write chunk: failed to upload chunk 1 with 5242880 bytes: RequestCanceled: request context canceled
caused by: context canceled
2023/09/20 10:32:58 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 4/205 failed: multi-thread copy: failed to write chunk: failed to upload chunk 4 with 5242880 bytes: RequestCanceled: request context canceled
caused by: context canceled
2023/09/20 10:32:58 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 5/205 failed: multi-thread copy: failed to write chunk: failed to upload chunk 5 with 5242880 bytes: RequestCanceled: request context canceled
caused by: context canceled
2023/09/20 10:32:58 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: cancelling transfer on exit
2023/09/20 10:32:58 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: abort failed: failed to abort multipart upload "2~l77PBr2f0GKMNB85FmER2x1NHvCXjGO": NoSuchUpload:
        status code: 404, request id: tx0000036ee33ddc56653d0-00650b029a-48ad0726-default, host id:
2023/09/20 10:32:58 ERROR : ap-prod-web-01.tgz.part-in: Failed to copy: multi-thread copy: failed to write chunk: failed to upload chunk 2 with 5242880 bytes: NoSuchUpload:
        status code: 404, request id: tx00000a9310d970224ed62-00650b029a-487043e8-default, host id:
2023/09/20 10:32:58 ERROR : Attempt 2/3 failed with 1 errors and: multi-thread copy: failed to write chunk: failed to upload chunk 2 with 5242880 bytes: NoSuchUpload:
        status code: 404, request id: tx00000a9310d970224ed62-00650b029a-487043e8-default, host id:
2023/09/20 10:32:58 DEBUG : ap-prod-web-01.tgz.part-in: Need to transfer - File not found at Destination
2023/09/20 10:33:03 DEBUG : ap-prod-web-01.tgz.part-in: open chunk writer: started multipart upload: 2~51ClppUcg0AERQOL3P2rmf9HUdAxo8o
2023/09/20 10:33:03 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 4
2023/09/20 10:33:03 DEBUG : ap-prod-web-01.tgz.part-in: Starting multi-thread copy with 205 chunks of size 5Mi with 4 parallel streams
2023/09/20 10:33:03 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 4/205 (15728640-20971520) size 5Mi starting
2023/09/20 10:33:03 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 1/205 (0-5242880) size 5Mi starting
2023/09/20 10:33:03 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 2/205 (5242880-10485760) size 5Mi starting
2023/09/20 10:33:03 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 3/205 (10485760-15728640) size 5Mi starting
2023/09/20 10:33:04 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 2/205 failed: multi-thread copy: failed to write chunk: failed to upload chunk 2 with 5242880 bytes: NoSuchUpload:
        status code: 404, request id: tx000006bf4d71afedab5c8-00650b029f-47ebb6f3-default, host id:
2023/09/20 10:33:04 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 5/205 (20971520-26214400) size 5Mi starting
2023/09/20 10:33:04 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 3/205 failed: multi-thread copy: failed to write chunk: failed to upload chunk 3 with 5242880 bytes: RequestCanceled: request context canceled
caused by: context canceled
2023/09/20 10:33:04 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 4/205 failed: multi-thread copy: failed to write chunk: failed to upload chunk 4 with 5242880 bytes: RequestCanceled: request context canceled
caused by: context canceled
2023/09/20 10:33:04 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 1/205 failed: multi-thread copy: failed to write chunk: failed to upload chunk 1 with 5242880 bytes: RequestCanceled: request context canceled
caused by: context canceled
2023/09/20 10:33:04 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 5/205 failed: multi-thread copy: failed to write chunk: failed to upload chunk 5 with 5242880 bytes: RequestCanceled: request context canceled
caused by: context canceled
2023/09/20 10:33:04 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: cancelling transfer on exit
2023/09/20 10:33:04 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: abort failed: failed to abort multipart upload "2~51ClppUcg0AERQOL3P2rmf9HUdAxo8o": NoSuchUpload:
        status code: 404, request id: tx000002c80b4a239a3cdc5-00650b02a0-483c497e-default, host id:
2023/09/20 10:33:04 ERROR : ap-prod-web-01.tgz.part-in: Failed to copy: multi-thread copy: failed to write chunk: failed to upload chunk 2 with 5242880 bytes: NoSuchUpload:
        status code: 404, request id: tx000006bf4d71afedab5c8-00650b029f-47ebb6f3-default, host id:
2023/09/20 10:33:04 ERROR : Attempt 3/3 failed with 1 errors and: multi-thread copy: failed to write chunk: failed to upload chunk 2 with 5242880 bytes: NoSuchUpload:
        status code: 404, request id: tx000006bf4d71afedab5c8-00650b029f-47ebb6f3-default, host id:
2023/09/20 10:33:04 INFO  :
Transferred:       26.188 MiB / 26.188 MiB, 100%, 1.032 MiB/s, ETA 0s
Errors:                 1 (retrying may help)
Elapsed time:        18.9s

2023/09/20 10:33:04 DEBUG : 9 go routines active
2023/09/20 10:33:04 Failed to copy: multi-thread copy: failed to write chunk: failed to upload chunk 2 with 5242880 bytes: NoSuchUpload:
        status code: 404, request id: tx000006bf4d71afedab5c8-00650b029f-47ebb6f3-default, host id:
5

what is the exact size of the file?

rclone did make some significant changes recently.
might test --multi-thread-streams=0 --s3-upload-concurrency=0

for a deeper look, -vv --dump=headers

6

This looks like a problem with the multipart upload code that was re-written in v1.64

Can you try v1.63.1 - I'd expect that to work if that is the case.

We don't currently have an S3/Ceph based provider in the integration tests (a bit of an oversight). We used to test with digital ocean but the integration tests triggered some awful rate limiting which meant they never completed.

If you could do a test which fails using -vv --dump headers that would be very useful.

7

Here it is with --dump headers (in 2 parts):

rclone -vv --dump headers copy /mnt/tmp-ap-backup2/bhg/ap-prod-web-01.tgz.part-in linodes3:/AP-prod-web-01/
2023/09/20 11:34:19 DEBUG : rclone: Version "v1.64.0" starting with parameters ["rclone" "-vv" "--dump" "headers" "copy" "/mnt/tmp-ap-backup2/bhg/ap-prod-web-01.tgz.part-in" "linodes3:/AP-prod-web-01/"]
2023/09/20 11:34:19 DEBUG : Creating backend with remote "/mnt/tmp-ap-backup2/bhg/ap-prod-web-01.tgz.part-in"
2023/09/20 11:34:19 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2023/09/20 11:34:19 DEBUG : fs cache: adding new entry for parent of "/mnt/tmp-ap-backup2/bhg/ap-prod-web-01.tgz.part-in", "/mnt/tmp-ap-backup2/bhg"
2023/09/20 11:34:19 DEBUG : Creating backend with remote "linodes3:/AP-prod-web-01/"
2023/09/20 11:34:19 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2023/09/20 11:34:19 DEBUG : Resolving service "s3" region "us-east-1"
2023/09/20 11:34:19 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2023/09/20 11:34:19 DEBUG : fs cache: renaming cache item "linodes3:/AP-prod-web-01/" to be canonical "linodes3:AP-prod-web-01"
2023/09/20 11:34:19 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/09/20 11:34:19 DEBUG : HTTP REQUEST (req 0xc0007ae900)
2023/09/20 11:34:19 DEBUG : HEAD /AP-prod-web-01/ap-prod-web-01.tgz.part-in HTTP/1.1
Host: server-backups.us-east-1.linodeobjects.com
User-Agent: rclone/v1.64.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20230920T153419Z

2023/09/20 11:34:19 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/09/20 11:34:19 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/09/20 11:34:19 DEBUG : HTTP RESPONSE (req 0xc0007ae900)
2023/09/20 11:34:19 DEBUG : HTTP/1.1 404 Not Found
Content-Length: 229
Accept-Ranges: bytes
Connection: keep-alive
Content-Type: application/xml
Date: Wed, 20 Sep 2023 15:34:19 GMT
X-Amz-Request-Id: tx00000393b9a2a8b1f0dc9-00650b10fb-48aaf8b1-default

2023/09/20 11:34:19 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/09/20 11:34:19 DEBUG : ap-prod-web-01.tgz.part-in: Need to transfer - File not found at Destination
2023/09/20 11:34:19 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/09/20 11:34:19 DEBUG : HTTP REQUEST (req 0xc0009fa400)
2023/09/20 11:34:19 DEBUG : PUT /AP-prod-web-01 HTTP/1.1
Host: server-backups.us-east-1.linodeobjects.com
User-Agent: rclone/v1.64.0
Content-Length: 0
Authorization: XXXX
X-Amz-Acl: private
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20230920T153419Z
Accept-Encoding: gzip

2023/09/20 11:34:19 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/09/20 11:34:19 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/09/20 11:34:19 DEBUG : HTTP RESPONSE (req 0xc0009fa400)
2023/09/20 11:34:19 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Accept-Ranges: bytes
Connection: keep-alive
Date: Wed, 20 Sep 2023 15:34:19 GMT
Etag: "d41d8cd98f00b204e9800998ecf8427e"
X-Amz-Request-Id: tx00000911d7127836c4c4f-00650b10fb-48aac703-default

2023/09/20 11:34:19 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/09/20 11:34:19 INFO  : S3 bucket AP-prod-web-01: Bucket "AP-prod-web-01" created with ACL "private"
2023/09/20 11:34:25 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/09/20 11:34:25 DEBUG : HTTP REQUEST (req 0xc0007ae400)
2023/09/20 11:34:25 DEBUG : POST /AP-prod-web-01/ap-prod-web-01.tgz.part-in?uploads= HTTP/1.1
Host: server-backups.us-east-1.linodeobjects.com
User-Agent: rclone/v1.64.0
Content-Length: 0
Authorization: XXXX
Content-Type: application/octet-stream
X-Amz-Acl: private
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20230920T153425Z
X-Amz-Meta-Md5chksum: 0sRoOTUwTX2CCkNKRc5RTA==
X-Amz-Meta-Mtime: 1695128550.932673007
Accept-Encoding: gzip

2023/09/20 11:34:25 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/09/20 11:34:25 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/09/20 11:34:25 DEBUG : HTTP RESPONSE (req 0xc0007ae400)
2023/09/20 11:34:25 DEBUG : HTTP/1.1 200 OK
Content-Length: 286
Connection: keep-alive
Content-Type: application/xml
Date: Wed, 20 Sep 2023 15:34:25 GMT
X-Amz-Request-Id: tx000008bf827a972098e9c-00650b1101-48aaf1ff-default

2023/09/20 11:34:25 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/09/20 11:34:25 DEBUG : ap-prod-web-01.tgz.part-in: open chunk writer: started multipart upload: 2~TvA-ab5G55Y5IQGWn2m4HRSrCHmezGZ
2023/09/20 11:34:25 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 4
2023/09/20 11:34:25 DEBUG : ap-prod-web-01.tgz.part-in: Starting multi-thread copy with 205 chunks of size 5Mi with 4 parallel streams
2023/09/20 11:34:25 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 4/205 (15728640-20971520) size 5Mi starting
2023/09/20 11:34:25 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 1/205 (0-5242880) size 5Mi starting
2023/09/20 11:34:25 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 3/205 (10485760-15728640) size 5Mi starting
2023/09/20 11:34:25 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 2/205 (5242880-10485760) size 5Mi starting
2023/09/20 11:34:25 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/09/20 11:34:25 DEBUG : HTTP REQUEST (req 0xc0009fa200)
2023/09/20 11:34:25 DEBUG : PUT /server-backups/AP-prod-web-01/ap-prod-web-01.tgz.part-in?partNumber=1&uploadId=2~TvA-ab5G55Y5IQGWn2m4HRSrCHmezGZ HTTP/1.1
Host: server-backups.us-east-1.linodeobjects.com
User-Agent: rclone/v1.64.0
Content-Length: 5242880
Authorization: XXXX
Content-Md5: UGFvnCA639NudNiL9GYi2g==
Expect: 100-continue
X-Amz-Content-Sha256: f7ec08073786e66676ea11b3ac6b16edd4ae088f9df450f61ee324b5e9c9a6d6
X-Amz-Date: 20230920T153425Z
Accept-Encoding: gzip

2023/09/20 11:34:25 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/09/20 11:34:25 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/09/20 11:34:25 DEBUG : HTTP REQUEST (req 0xc000904500)
2023/09/20 11:34:25 DEBUG : PUT /server-backups/AP-prod-web-01/ap-prod-web-01.tgz.part-in?partNumber=4&uploadId=2~TvA-ab5G55Y5IQGWn2m4HRSrCHmezGZ HTTP/1.1
Host: server-backups.us-east-1.linodeobjects.com
User-Agent: rclone/v1.64.0
Content-Length: 5242880
Authorization: XXXX
Content-Md5: sIrEX6DJslBF7o2cGRU0rw==
Expect: 100-continue
X-Amz-Content-Sha256: b17bded241750f8148a6f7ded1b04572ceeaffed906cfe8faf1b037dfa4fa594
X-Amz-Date: 20230920T153425Z
Accept-Encoding: gzip

2023/09/20 11:34:25 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/09/20 11:34:25 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/09/20 11:34:25 DEBUG : HTTP REQUEST (req 0xc0007ae200)
2023/09/20 11:34:25 DEBUG : PUT /server-backups/AP-prod-web-01/ap-prod-web-01.tgz.part-in?partNumber=3&uploadId=2~TvA-ab5G55Y5IQGWn2m4HRSrCHmezGZ HTTP/1.1
Host: server-backups.us-east-1.linodeobjects.com
User-Agent: rclone/v1.64.0
Content-Length: 5242880
Authorization: XXXX
Content-Md5: q+KIetpQFyUTOq3M3vThfQ==
Expect: 100-continue
X-Amz-Content-Sha256: 0010cdf014b805221c6ebe93a8e0b9ddd74dfbcbb08b154d038a355219d71484
X-Amz-Date: 20230920T153425Z
Accept-Encoding: gzip

2023/09/20 11:34:25 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/09/20 11:34:25 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/09/20 11:34:25 DEBUG : HTTP REQUEST (req 0xc0001f8a00)
2023/09/20 11:34:25 DEBUG : PUT /server-backups/AP-prod-web-01/ap-prod-web-01.tgz.part-in?partNumber=2&uploadId=2~TvA-ab5G55Y5IQGWn2m4HRSrCHmezGZ HTTP/1.1
Host: server-backups.us-east-1.linodeobjects.com
User-Agent: rclone/v1.64.0
Content-Length: 5242880
Authorization: XXXX
Content-Md5: olgMGfu2cXHgwqtdjTJf7w==
Expect: 100-continue
X-Amz-Content-Sha256: e9dd81cd9559bba78edb9b6dead5e6a706525ef77cc548e4af2f82e3bcec54cd
X-Amz-Date: 20230920T153425Z
Accept-Encoding: gzip

2023/09/20 11:34:25 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/09/20 11:34:25 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/09/20 11:34:25 DEBUG : HTTP RESPONSE (req 0xc0009fa200)
2023/09/20 11:34:25 DEBUG : HTTP/1.1 404 Not Found
Content-Length: 232
Accept-Ranges: bytes
Connection: keep-alive
Content-Type: application/xml
Date: Wed, 20 Sep 2023 15:34:25 GMT
X-Amz-Request-Id: tx000003a6191b34dc3b791-00650b1101-48ace50e-default

2023/09/20 11:34:25 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/09/20 11:34:25 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 1/205 failed: multi-thread copy: failed to write chunk: failed to upload chunk 1 with 5242880 bytes: NoSuchUpload:
        status code: 404, request id: tx000003a6191b34dc3b791-00650b1101-48ace50e-default, host id:
2023/09/20 11:34:25 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 5/205 (20971520-26214400) size 5Mi starting
2023/09/20 11:34:25 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/09/20 11:34:25 DEBUG : HTTP RESPONSE (req 0xc0007ae200)
2023/09/20 11:34:25 DEBUG : Error: context canceled
2023/09/20 11:34:25 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/09/20 11:34:25 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 3/205 failed: multi-thread copy: failed to write chunk: failed to upload chunk 3 with 5242880 bytes: RequestCanceled: request context canceled
caused by: context canceled
2023/09/20 11:34:25 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/09/20 11:34:25 DEBUG : HTTP RESPONSE (req 0xc000904500)
2023/09/20 11:34:25 DEBUG : Error: context canceled
2023/09/20 11:34:25 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/09/20 11:34:25 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 4/205 failed: multi-thread copy: failed to write chunk: failed to upload chunk 4 with 5242880 bytes: RequestCanceled: request context canceled
caused by: context canceled
2023/09/20 11:34:25 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/09/20 11:34:25 DEBUG : HTTP RESPONSE (req 0xc0001f8a00)
2023/09/20 11:34:25 DEBUG : Error: context canceled
2023/09/20 11:34:25 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/09/20 11:34:25 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 2/205 failed: multi-thread copy: failed to write chunk: failed to upload chunk 2 with 5242880 bytes: RequestCanceled: request context canceled
caused by: context canceled
2023/09/20 11:34:25 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 5/205 failed: multi-thread copy: failed to write chunk: failed to upload chunk 5 with 5242880 bytes: RequestCanceled: request context canceled
caused by: context canceled
2023/09/20 11:34:25 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/09/20 11:34:25 DEBUG : HTTP REQUEST (req 0xc0009fb600)
2023/09/20 11:34:25 DEBUG : POST /server-backups/AP-prod-web-01/ap-prod-web-01.tgz.part-in?uploadId=2~TvA-ab5G55Y5IQGWn2m4HRSrCHmezGZ HTTP/1.1
Host: server-backups.us-east-1.linodeobjects.com
User-Agent: rclone/v1.64.0
Content-Length: 99
Authorization: XXXX
X-Amz-Content-Sha256: 4792f8bb2fabe35af9a3a5d2d7b77f044b36b7f64b1662ba148892d4bca11a49
X-Amz-Date: 20230920T153425Z
Accept-Encoding: gzip

2023/09/20 11:34:25 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/09/20 11:34:25 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/09/20 11:34:25 DEBUG : HTTP RESPONSE (req 0xc0009fb600)
2023/09/20 11:34:25 DEBUG : HTTP/1.1 400 Bad Request
Content-Length: 232
Accept-Ranges: bytes
Connection: keep-alive
Content-Type: application/xml
Date: Wed, 20 Sep 2023 15:34:25 GMT
X-Amz-Request-Id: tx000005af5ac3e82292976-00650b1101-48aac919-default

2023/09/20 11:34:25 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/09/20 11:34:25 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: cancelling transfer on exit
2023/09/20 11:34:25 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/09/20 11:34:25 DEBUG : HTTP REQUEST (req 0xc0009fbf00)
2023/09/20 11:34:25 DEBUG : DELETE /server-backups/AP-prod-web-01/ap-prod-web-01.tgz.part-in?uploadId=2~TvA-ab5G55Y5IQGWn2m4HRSrCHmezGZ HTTP/1.1
Host: server-backups.us-east-1.linodeobjects.com
User-Agent: rclone/v1.64.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20230920T153425Z
Accept-Encoding: gzip

2023/09/20 11:34:25 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/09/20 11:34:25 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/09/20 11:34:25 DEBUG : HTTP RESPONSE (req 0xc0009fbf00)
2023/09/20 11:34:25 DEBUG : HTTP/1.1 404 Not Found
Content-Length: 232
Accept-Ranges: bytes
Connection: keep-alive
Content-Type: application/xml
Date: Wed, 20 Sep 2023 15:34:25 GMT
X-Amz-Request-Id: tx0000058e6dbcf55381a44-00650b1101-487043e8-default

2023/09/20 11:34:25 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/09/20 11:34:25 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: abort failed: failed to abort multipart upload "2~TvA-ab5G55Y5IQGWn2m4HRSrCHmezGZ": NoSuchUpload:
        status code: 404, request id: tx0000058e6dbcf55381a44-00650b1101-487043e8-default, host id:
2023/09/20 11:34:25 ERROR : ap-prod-web-01.tgz.part-in: Failed to copy: multi-thread copy: failed to write chunk: failed to upload chunk 1 with 5242880 bytes: NoSuchUpload:
        status code: 404, request id: tx000003a6191b34dc3b791-00650b1101-48ace50e-default, host id:
2023/09/20 11:34:25 ERROR : Attempt 1/3 failed with 1 errors and: multi-thread copy: failed to write chunk: failed to upload chunk 1 with 5242880 bytes: NoSuchUpload:
        status code: 404, request id: tx000003a6191b34dc3b791-00650b1101-48ace50e-default, host id:
2023/09/20 11:34:25 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/09/20 11:34:25 DEBUG : HTTP REQUEST (req 0xc00093c000)
2023/09/20 11:34:25 DEBUG : HEAD /AP-prod-web-01/ap-prod-web-01.tgz.part-in HTTP/1.1
Host: server-backups.us-east-1.linodeobjects.com
User-Agent: rclone/v1.64.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20230920T153425Z

2023/09/20 11:34:25 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/09/20 11:34:25 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/09/20 11:34:25 DEBUG : HTTP RESPONSE (req 0xc00093c000)
2023/09/20 11:34:25 DEBUG : HTTP/1.1 404 Not Found
Content-Length: 229
Accept-Ranges: bytes
Connection: keep-alive
Content-Type: application/xml
Date: Wed, 20 Sep 2023 15:34:25 GMT
X-Amz-Request-Id: tx00000bf7fc56dbcf92d6a-00650b1101-486cef07-default

2023/09/20 11:34:25 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/09/20 11:34:25 DEBUG : ap-prod-web-01.tgz.part-in: Need to transfer - File not found at Destination
2023/09/20 11:34:31 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/09/20 11:34:31 DEBUG : HTTP REQUEST (req 0xc000d30200)
2023/09/20 11:34:31 DEBUG : POST /AP-prod-web-01/ap-prod-web-01.tgz.part-in?uploads= HTTP/1.1
Host: server-backups.us-east-1.linodeobjects.com
User-Agent: rclone/v1.64.0
Content-Length: 0
Authorization: XXXX
Content-Type: application/octet-stream
X-Amz-Acl: private
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20230920T153431Z
X-Amz-Meta-Md5chksum: 0sRoOTUwTX2CCkNKRc5RTA==
X-Amz-Meta-Mtime: 1695128550.932673007
Accept-Encoding: gzip

2023/09/20 11:34:31 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/09/20 11:34:31 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/09/20 11:34:31 DEBUG : HTTP RESPONSE (req 0xc000d30200)
2023/09/20 11:34:31 DEBUG : HTTP/1.1 200 OK
Content-Length: 286
Connection: keep-alive
Content-Type: application/xml
Date: Wed, 20 Sep 2023 15:34:31 GMT
X-Amz-Request-Id: tx00000069479ea614e4409-00650b1107-485dec0d-default

2023/09/20 11:34:31 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/09/20 11:34:31 DEBUG : ap-prod-web-01.tgz.part-in: open chunk writer: started multipart upload: 2~-SuTky-eTn7PsDDXrfoZXULDSiuYFNz
2023/09/20 11:34:31 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 4
2023/09/20 11:34:31 DEBUG : ap-prod-web-01.tgz.part-in: Starting multi-thread copy with 205 chunks of size 5Mi with 4 parallel streams
2023/09/20 11:34:31 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 4/205 (15728640-20971520) size 5Mi starting
2023/09/20 11:34:31 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 1/205 (0-5242880) size 5Mi starting
2023/09/20 11:34:31 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 2/205 (5242880-10485760) size 5Mi starting
2023/09/20 11:34:31 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 3/205 (10485760-15728640) size 5Mi starting
8

part 2/2:

2023/09/20 11:34:31 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/09/20 11:34:31 DEBUG : HTTP REQUEST (req 0xc0007aeb00)
2023/09/20 11:34:31 DEBUG : PUT /server-backups/AP-prod-web-01/ap-prod-web-01.tgz.part-in?partNumber=2&uploadId=2~-SuTky-eTn7PsDDXrfoZXULDSiuYFNz HTTP/1.1
Host: server-backups.us-east-1.linodeobjects.com
User-Agent: rclone/v1.64.0
Content-Length: 5242880
Authorization: XXXX
Content-Md5: olgMGfu2cXHgwqtdjTJf7w==
Expect: 100-continue
X-Amz-Content-Sha256: e9dd81cd9559bba78edb9b6dead5e6a706525ef77cc548e4af2f82e3bcec54cd
X-Amz-Date: 20230920T153431Z
Accept-Encoding: gzip

2023/09/20 11:34:31 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/09/20 11:34:31 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/09/20 11:34:31 DEBUG : HTTP REQUEST (req 0xc00061e100)
2023/09/20 11:34:31 DEBUG : PUT /server-backups/AP-prod-web-01/ap-prod-web-01.tgz.part-in?partNumber=3&uploadId=2~-SuTky-eTn7PsDDXrfoZXULDSiuYFNz HTTP/1.1
Host: server-backups.us-east-1.linodeobjects.com
User-Agent: rclone/v1.64.0
Content-Length: 5242880
Authorization: XXXX
Content-Md5: q+KIetpQFyUTOq3M3vThfQ==
Expect: 100-continue
X-Amz-Content-Sha256: 0010cdf014b805221c6ebe93a8e0b9ddd74dfbcbb08b154d038a355219d71484
X-Amz-Date: 20230920T153431Z
Accept-Encoding: gzip

2023/09/20 11:34:31 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/09/20 11:34:31 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/09/20 11:34:31 DEBUG : HTTP RESPONSE (req 0xc0007aeb00)
2023/09/20 11:34:31 DEBUG : HTTP/1.1 404 Not Found
Content-Length: 232
Accept-Ranges: bytes
Connection: keep-alive
Content-Type: application/xml
Date: Wed, 20 Sep 2023 15:34:31 GMT
X-Amz-Request-Id: tx00000802ed8e386896b49-00650b1107-4889dfd6-default

2023/09/20 11:34:31 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/09/20 11:34:31 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/09/20 11:34:31 DEBUG : HTTP REQUEST (req 0xc000b32200)
2023/09/20 11:34:31 DEBUG : PUT /server-backups/AP-prod-web-01/ap-prod-web-01.tgz.part-in?partNumber=4&uploadId=2~-SuTky-eTn7PsDDXrfoZXULDSiuYFNz HTTP/1.1
Host: server-backups.us-east-1.linodeobjects.com
User-Agent: rclone/v1.64.0
Content-Length: 5242880
Authorization: XXXX
Content-Md5: sIrEX6DJslBF7o2cGRU0rw==
Expect: 100-continue
X-Amz-Content-Sha256: b17bded241750f8148a6f7ded1b04572ceeaffed906cfe8faf1b037dfa4fa594
X-Amz-Date: 20230920T153431Z
Accept-Encoding: gzip

2023/09/20 11:34:31 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/09/20 11:34:31 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/09/20 11:34:31 DEBUG : HTTP REQUEST (req 0xc000d30600)
2023/09/20 11:34:31 DEBUG : PUT /server-backups/AP-prod-web-01/ap-prod-web-01.tgz.part-in?partNumber=1&uploadId=2~-SuTky-eTn7PsDDXrfoZXULDSiuYFNz HTTP/1.1
Host: server-backups.us-east-1.linodeobjects.com
User-Agent: rclone/v1.64.0
Content-Length: 5242880
Authorization: XXXX
Content-Md5: UGFvnCA639NudNiL9GYi2g==
Expect: 100-continue
X-Amz-Content-Sha256: f7ec08073786e66676ea11b3ac6b16edd4ae088f9df450f61ee324b5e9c9a6d6
X-Amz-Date: 20230920T153431Z
Accept-Encoding: gzip

2023/09/20 11:34:31 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/09/20 11:34:31 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/09/20 11:34:31 DEBUG : HTTP RESPONSE (req 0xc00061e100)
2023/09/20 11:34:31 DEBUG : Error: http: server closed idle connection
2023/09/20 11:34:31 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/09/20 11:34:31 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 2/205 failed: multi-thread copy: failed to write chunk: failed to upload chunk 2 with 5242880 bytes: NoSuchUpload:
        status code: 404, request id: tx00000802ed8e386896b49-00650b1107-4889dfd6-default, host id:
2023/09/20 11:34:31 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/09/20 11:34:31 DEBUG : HTTP RESPONSE (req 0xc000b32200)
2023/09/20 11:34:31 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 3/205 failed: multi-thread copy: failed to write chunk: failed to upload chunk 3 with 5242880 bytes: RequestCanceled: request context canceled
caused by: context canceled
2023/09/20 11:34:31 DEBUG : Error: context canceled
2023/09/20 11:34:31 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/09/20 11:34:31 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 5/205 (20971520-26214400) size 5Mi starting
2023/09/20 11:34:31 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 4/205 failed: multi-thread copy: failed to write chunk: failed to upload chunk 4 with 5242880 bytes: RequestCanceled: request context canceled
caused by: context canceled
2023/09/20 11:34:31 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/09/20 11:34:31 DEBUG : HTTP RESPONSE (req 0xc000d30600)
2023/09/20 11:34:31 DEBUG : Error: context canceled
2023/09/20 11:34:31 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/09/20 11:34:31 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 1/205 failed: multi-thread copy: failed to write chunk: failed to upload chunk 1 with 5242880 bytes: RequestCanceled: request context canceled
caused by: context canceled
2023/09/20 11:34:31 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 5/205 failed: multi-thread copy: failed to write chunk: failed to upload chunk 5 with 5242880 bytes: RequestCanceled: request context canceled
caused by: context canceled
2023/09/20 11:34:31 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/09/20 11:34:31 DEBUG : HTTP REQUEST (req 0xc000d30300)
2023/09/20 11:34:31 DEBUG : POST /server-backups/AP-prod-web-01/ap-prod-web-01.tgz.part-in?uploadId=2~-SuTky-eTn7PsDDXrfoZXULDSiuYFNz HTTP/1.1
Host: server-backups.us-east-1.linodeobjects.com
User-Agent: rclone/v1.64.0
Content-Length: 99
Authorization: XXXX
X-Amz-Content-Sha256: 4792f8bb2fabe35af9a3a5d2d7b77f044b36b7f64b1662ba148892d4bca11a49
X-Amz-Date: 20230920T153431Z
Accept-Encoding: gzip

2023/09/20 11:34:31 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/09/20 11:34:31 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/09/20 11:34:31 DEBUG : HTTP RESPONSE (req 0xc000d30300)
2023/09/20 11:34:31 DEBUG : HTTP/1.1 400 Bad Request
Content-Length: 232
Accept-Ranges: bytes
Connection: keep-alive
Content-Type: application/xml
Date: Wed, 20 Sep 2023 15:34:31 GMT
X-Amz-Request-Id: tx00000572b5393de4c44ca-00650b1107-48b08034-default

2023/09/20 11:34:31 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/09/20 11:34:31 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: cancelling transfer on exit
2023/09/20 11:34:31 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/09/20 11:34:31 DEBUG : HTTP REQUEST (req 0xc0007ae600)
2023/09/20 11:34:31 DEBUG : DELETE /server-backups/AP-prod-web-01/ap-prod-web-01.tgz.part-in?uploadId=2~-SuTky-eTn7PsDDXrfoZXULDSiuYFNz HTTP/1.1
Host: server-backups.us-east-1.linodeobjects.com
User-Agent: rclone/v1.64.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20230920T153431Z
Accept-Encoding: gzip

2023/09/20 11:34:31 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/09/20 11:34:31 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/09/20 11:34:31 DEBUG : HTTP RESPONSE (req 0xc0007ae600)
2023/09/20 11:34:31 DEBUG : HTTP/1.1 404 Not Found
Content-Length: 232
Accept-Ranges: bytes
Connection: keep-alive
Content-Type: application/xml
Date: Wed, 20 Sep 2023 15:34:31 GMT
X-Amz-Request-Id: tx00000bd74cc4e885209b6-00650b1107-483c497e-default

2023/09/20 11:34:31 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/09/20 11:34:31 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: abort failed: failed to abort multipart upload "2~-SuTky-eTn7PsDDXrfoZXULDSiuYFNz": NoSuchUpload:
        status code: 404, request id: tx00000bd74cc4e885209b6-00650b1107-483c497e-default, host id:
2023/09/20 11:34:31 ERROR : ap-prod-web-01.tgz.part-in: Failed to copy: multi-thread copy: failed to write chunk: failed to upload chunk 2 with 5242880 bytes: NoSuchUpload:
        status code: 404, request id: tx00000802ed8e386896b49-00650b1107-4889dfd6-default, host id:
2023/09/20 11:34:31 ERROR : Attempt 2/3 failed with 1 errors and: multi-thread copy: failed to write chunk: failed to upload chunk 2 with 5242880 bytes: NoSuchUpload:
        status code: 404, request id: tx00000802ed8e386896b49-00650b1107-4889dfd6-default, host id:
2023/09/20 11:34:31 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/09/20 11:34:31 DEBUG : HTTP REQUEST (req 0xc000d30e00)
2023/09/20 11:34:31 DEBUG : HEAD /AP-prod-web-01/ap-prod-web-01.tgz.part-in HTTP/1.1
Host: server-backups.us-east-1.linodeobjects.com
User-Agent: rclone/v1.64.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20230920T153431Z

2023/09/20 11:34:31 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/09/20 11:34:31 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/09/20 11:34:31 DEBUG : HTTP RESPONSE (req 0xc000d30e00)
2023/09/20 11:34:31 DEBUG : HTTP/1.1 404 Not Found
Content-Length: 229
Accept-Ranges: bytes
Connection: keep-alive
Content-Type: application/xml
Date: Wed, 20 Sep 2023 15:34:31 GMT
X-Amz-Request-Id: tx00000785e230df7280fc1-00650b1107-48ab4662-default

2023/09/20 11:34:31 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/09/20 11:34:31 DEBUG : ap-prod-web-01.tgz.part-in: Need to transfer - File not found at Destination
2023/09/20 11:34:37 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/09/20 11:34:37 DEBUG : HTTP REQUEST (req 0xc000b32a00)
2023/09/20 11:34:37 DEBUG : POST /AP-prod-web-01/ap-prod-web-01.tgz.part-in?uploads= HTTP/1.1
Host: server-backups.us-east-1.linodeobjects.com
User-Agent: rclone/v1.64.0
Content-Length: 0
Authorization: XXXX
Content-Type: application/octet-stream
X-Amz-Acl: private
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20230920T153437Z
X-Amz-Meta-Md5chksum: 0sRoOTUwTX2CCkNKRc5RTA==
X-Amz-Meta-Mtime: 1695128550.932673007
Accept-Encoding: gzip

2023/09/20 11:34:37 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/09/20 11:34:37 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/09/20 11:34:37 DEBUG : HTTP RESPONSE (req 0xc000b32a00)
2023/09/20 11:34:37 DEBUG : HTTP/1.1 200 OK
Content-Length: 286
Connection: keep-alive
Content-Type: application/xml
Date: Wed, 20 Sep 2023 15:34:37 GMT
X-Amz-Request-Id: tx000000ffe44c5bcb143a4-00650b110d-48ab036a-default

2023/09/20 11:34:37 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/09/20 11:34:37 DEBUG : ap-prod-web-01.tgz.part-in: open chunk writer: started multipart upload: 2~JfSbU0St4oVE8FIAc3pS-LWfvupUdWr
2023/09/20 11:34:37 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 4
2023/09/20 11:34:37 DEBUG : ap-prod-web-01.tgz.part-in: Starting multi-thread copy with 205 chunks of size 5Mi with 4 parallel streams
2023/09/20 11:34:37 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 4/205 (15728640-20971520) size 5Mi starting
2023/09/20 11:34:37 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 1/205 (0-5242880) size 5Mi starting
2023/09/20 11:34:37 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 2/205 (5242880-10485760) size 5Mi starting
2023/09/20 11:34:37 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 3/205 (10485760-15728640) size 5Mi starting
2023/09/20 11:34:37 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/09/20 11:34:37 DEBUG : HTTP REQUEST (req 0xc00093c200)
2023/09/20 11:34:37 DEBUG : PUT /server-backups/AP-prod-web-01/ap-prod-web-01.tgz.part-in?partNumber=3&uploadId=2~JfSbU0St4oVE8FIAc3pS-LWfvupUdWr HTTP/1.1
Host: server-backups.us-east-1.linodeobjects.com
User-Agent: rclone/v1.64.0
Content-Length: 5242880
Authorization: XXXX
Content-Md5: q+KIetpQFyUTOq3M3vThfQ==
Expect: 100-continue
X-Amz-Content-Sha256: 0010cdf014b805221c6ebe93a8e0b9ddd74dfbcbb08b154d038a355219d71484
X-Amz-Date: 20230920T153437Z
Accept-Encoding: gzip

2023/09/20 11:34:37 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/09/20 11:34:37 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/09/20 11:34:37 DEBUG : HTTP REQUEST (req 0xc0007af000)
2023/09/20 11:34:37 DEBUG : PUT /server-backups/AP-prod-web-01/ap-prod-web-01.tgz.part-in?partNumber=2&uploadId=2~JfSbU0St4oVE8FIAc3pS-LWfvupUdWr HTTP/1.1
Host: server-backups.us-east-1.linodeobjects.com
User-Agent: rclone/v1.64.0
Content-Length: 5242880
Authorization: XXXX
Content-Md5: olgMGfu2cXHgwqtdjTJf7w==
Expect: 100-continue
X-Amz-Content-Sha256: e9dd81cd9559bba78edb9b6dead5e6a706525ef77cc548e4af2f82e3bcec54cd
X-Amz-Date: 20230920T153437Z
Accept-Encoding: gzip

2023/09/20 11:34:37 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/09/20 11:34:37 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/09/20 11:34:37 DEBUG : HTTP RESPONSE (req 0xc00093c200)
2023/09/20 11:34:37 DEBUG : HTTP/1.1 404 Not Found
Content-Length: 232
Accept-Ranges: bytes
Connection: keep-alive
Content-Type: application/xml
Date: Wed, 20 Sep 2023 15:34:37 GMT
X-Amz-Request-Id: tx00000c178aad12c4daa5a-00650b110d-486cef07-default

2023/09/20 11:34:37 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/09/20 11:34:37 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/09/20 11:34:37 DEBUG : HTTP REQUEST (req 0xc0007b8300)
2023/09/20 11:34:37 DEBUG : PUT /server-backups/AP-prod-web-01/ap-prod-web-01.tgz.part-in?partNumber=4&uploadId=2~JfSbU0St4oVE8FIAc3pS-LWfvupUdWr HTTP/1.1
Host: server-backups.us-east-1.linodeobjects.com
User-Agent: rclone/v1.64.0
Content-Length: 5242880
Authorization: XXXX
Content-Md5: sIrEX6DJslBF7o2cGRU0rw==
Expect: 100-continue
X-Amz-Content-Sha256: b17bded241750f8148a6f7ded1b04572ceeaffed906cfe8faf1b037dfa4fa594
X-Amz-Date: 20230920T153437Z
Accept-Encoding: gzip

2023/09/20 11:34:37 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/09/20 11:34:37 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/09/20 11:34:37 DEBUG : HTTP RESPONSE (req 0xc0007af000)
2023/09/20 11:34:37 DEBUG : Error: http: server closed idle connection
2023/09/20 11:34:37 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/09/20 11:34:37 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 3/205 failed: multi-thread copy: failed to write chunk: failed to upload chunk 3 with 5242880 bytes: NoSuchUpload:
        status code: 404, request id: tx00000c178aad12c4daa5a-00650b110d-486cef07-default, host id:
2023/09/20 11:34:37 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 5/205 (20971520-26214400) size 5Mi starting
2023/09/20 11:34:37 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 2/205 failed: multi-thread copy: failed to write chunk: failed to upload chunk 2 with 5242880 bytes: RequestCanceled: request context canceled
caused by: context canceled
2023/09/20 11:34:37 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/09/20 11:34:37 DEBUG : HTTP RESPONSE (req 0xc0007b8300)
2023/09/20 11:34:37 DEBUG : Error: context canceled
2023/09/20 11:34:37 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/09/20 11:34:37 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 4/205 failed: multi-thread copy: failed to write chunk: failed to upload chunk 4 with 5242880 bytes: RequestCanceled: request context canceled
caused by: context canceled
2023/09/20 11:34:37 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/09/20 11:34:37 DEBUG : HTTP REQUEST (req 0xc00061e600)
2023/09/20 11:34:37 DEBUG :
2023/09/20 11:34:37 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/09/20 11:34:37 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/09/20 11:34:37 DEBUG : HTTP RESPONSE (req 0xc00061e600)
2023/09/20 11:34:37 DEBUG : Error: context canceled
2023/09/20 11:34:37 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/09/20 11:34:37 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 1/205 failed: multi-thread copy: failed to write chunk: failed to upload chunk 1 with 5242880 bytes: RequestCanceled: request context canceled
caused by: context canceled
2023/09/20 11:34:37 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: chunk 5/205 failed: multi-thread copy: failed to write chunk: failed to upload chunk 5 with 5242880 bytes: RequestCanceled: request context canceled
caused by: context canceled
2023/09/20 11:34:37 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/09/20 11:34:37 DEBUG : HTTP REQUEST (req 0xc00061e400)
2023/09/20 11:34:37 DEBUG : POST /server-backups/AP-prod-web-01/ap-prod-web-01.tgz.part-in?uploadId=2~JfSbU0St4oVE8FIAc3pS-LWfvupUdWr HTTP/1.1
Host: server-backups.us-east-1.linodeobjects.com
User-Agent: rclone/v1.64.0
Content-Length: 99
Authorization: XXXX
X-Amz-Content-Sha256: 4792f8bb2fabe35af9a3a5d2d7b77f044b36b7f64b1662ba148892d4bca11a49
X-Amz-Date: 20230920T153437Z
Accept-Encoding: gzip

2023/09/20 11:34:37 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/09/20 11:34:37 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/09/20 11:34:37 DEBUG : HTTP RESPONSE (req 0xc00061e400)
2023/09/20 11:34:37 DEBUG : HTTP/1.1 400 Bad Request
Content-Length: 232
Accept-Ranges: bytes
Connection: keep-alive
Content-Type: application/xml
Date: Wed, 20 Sep 2023 15:34:37 GMT
X-Amz-Request-Id: tx00000fdf8a575a68f0cc1-00650b110d-48ad0726-default

2023/09/20 11:34:37 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/09/20 11:34:37 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: cancelling transfer on exit
2023/09/20 11:34:37 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/09/20 11:34:37 DEBUG : HTTP REQUEST (req 0xc0007ae500)
2023/09/20 11:34:37 DEBUG : DELETE /server-backups/AP-prod-web-01/ap-prod-web-01.tgz.part-in?uploadId=2~JfSbU0St4oVE8FIAc3pS-LWfvupUdWr HTTP/1.1
Host: server-backups.us-east-1.linodeobjects.com
User-Agent: rclone/v1.64.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20230920T153437Z
Accept-Encoding: gzip

2023/09/20 11:34:37 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/09/20 11:34:37 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/09/20 11:34:37 DEBUG : HTTP RESPONSE (req 0xc0007ae500)
2023/09/20 11:34:37 DEBUG : HTTP/1.1 404 Not Found
Content-Length: 232
Accept-Ranges: bytes
Connection: keep-alive
Content-Type: application/xml
Date: Wed, 20 Sep 2023 15:34:37 GMT
X-Amz-Request-Id: tx00000a5f3b7e22f8e84bd-00650b110d-482c098f-default

2023/09/20 11:34:37 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/09/20 11:34:37 DEBUG : ap-prod-web-01.tgz.part-in: multi-thread copy: abort failed: failed to abort multipart upload "2~JfSbU0St4oVE8FIAc3pS-LWfvupUdWr": NoSuchUpload:
        status code: 404, request id: tx00000a5f3b7e22f8e84bd-00650b110d-482c098f-default, host id:
2023/09/20 11:34:37 ERROR : ap-prod-web-01.tgz.part-in: Failed to copy: multi-thread copy: failed to write chunk: failed to upload chunk 3 with 5242880 bytes: NoSuchUpload:
        status code: 404, request id: tx00000c178aad12c4daa5a-00650b110d-486cef07-default, host id:
2023/09/20 11:34:37 ERROR : Attempt 3/3 failed with 1 errors and: multi-thread copy: failed to write chunk: failed to upload chunk 3 with 5242880 bytes: NoSuchUpload:
        status code: 404, request id: tx00000c178aad12c4daa5a-00650b110d-486cef07-default, host id:
2023/09/20 11:34:37 INFO  :
Transferred:       19.469 MiB / 19.469 MiB, 100%, 868.648 KiB/s, ETA 0s
Errors:                 1 (retrying may help)
Elapsed time:        18.0s

2023/09/20 11:34:37 DEBUG : 9 go routines active
2023/09/20 11:34:37 Failed to copy: multi-thread copy: failed to write chunk: failed to upload chunk 3 with 5242880 bytes: NoSuchUpload:
        status code: 404, request id: tx00000c178aad12c4daa5a-00650b110d-486cef07-default, host id:
9

Confirm that this issue does not occur in v1.63.1 and does occur in v1.64.0

10

File size is 1,073,741,824 bytes, but anything 200mb or bigger has the same issue. The largest size I got to work was 10mb so the threshold is somewhere between 10mb and 200mb.

I tried multiple settings for multi-thread-streams. Don't think I played with upload concurrency.

Since downgrading to 1.63.1 works, I need to move on with other fires to put out. But let me know if you want me to test something else.

11

I think I see the problem in your config

I set up a linode account to see if I could replicate the problem and it works fine for me.

The endpoint looks like this though

endpoint = eu-central-1.linodeobjects.com

So I think your endpoint should be

endpoint = us-east-1.linodeobjects.com

You need to name your bucket on the command line. I'm guessing that is AP-prod-web-01 so you'd use linodes3:AP-prod-web-01 to access that bucket (no / needed).

Can you give that a go?

12

Interesting, and thank you for the idea. If I was stuck, I'd be trying that in a heartbeat. The thing is, it does work, if I just go back one lower version to 1.63.1.

The instructions Akamai/Linode gives for the endpoint is https://[bucket name].[region].linodeobjects.com
The bucket is server-backups and the region is us-east-1
So I'm pretty sure the endpoint of https://server-backups.us-east-1.linodeobjects.com is correct.
AP-prod-web-01 is the top folder name.

It works exactly as expected with rclone 1.63.1. It just breaks with 1.64.0.
I've loaded three different versions side-by-side to test, and 1.61.1 works, 1.63.1 works, and 1.64.0 breaks. No difference in syntax.

13

Rclone automatically puts the bucket name on to the endpoint so you definitely shouldn't have it in your config.

14

Well dang, @ncw, that was indeed the issue. Looks like the Akamai instructions were wrong.

To clarify, 1.64.0 does work correctly if:
the endpoint in rclone.conf only defines the region (e.g., us-east-1.linodeobjects.com)
the command line specifies bucket name and folder(s), e.g.,
rclone copy [filespec] linodes3:/AP-web/backups

There may still be a bug related to the handling of the protocol (https) in the endpoint, maybe? Don't know why that would break between one version and the next.

But stripping that out of the config endpoint made that version work.

15

Great - I'm glad that works.

I'm not sure why your original config did work in v1.63 and didn't in v1.64. The cause will almost certainly be the re-write of the multipart uploading. The surprise is really why did it work at all in v1.63.

I should probably make a linode provider which configures the endpoints for linode specifically - that would stop the user confusion - or if anyone else would like to have a go?

16

I added a Linode provider here

v1.65.0-beta.7382.626bc2e75.fix-s3-linode on branch fix-s3-linode (uploaded in 15-30 mins)

I used it as an opportunity to document the process to make it easier for those who follow!

17

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.