Uploading large files to R2 (~ 250 MiB) with rclone causes signature errors

Help and Support
1

What is the problem you are having with rclone?

Run the command 'rclone version' and share the full output of the command.

rclone v1.59.2

  • os/version: ubuntu 20.04 (64 bit)
  • os/kernel: 5.15.0-46-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.18.6
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Cloudflare's R2

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy runs/1664306137-elite-chameleon/ r2:/checkpoints/elite-chameleon --progress

The rclone config contents with secrets removed.

[r2]
type = s3
provider = Other
access_key_id =
secret_access_key = 
endpoint =

A log from the command with the -vv flag

Note, by this point I'd managed to sync most of the files by running the above command multiple times:

2022/09/28 00:38:57 DEBUG : rclone: Version "v1.59.2" starting with parameters ["rclone" "copy" "runs/1664306137-elite-chameleon/" "r2:/checkpoints/elite-chameleon" "--progress" "-vv"]
2022/09/28 00:38:57 DEBUG : Creating backend with remote "runs/1664306137-elite-chameleon/"
2022/09/28 00:38:57 DEBUG : Using config file from "/home/ubuntu/.config/rclone/rclone.conf"
2022/09/28 00:38:57 DEBUG : fs cache: renaming cache item "runs/1664306137-elite-chameleon/" to be canonical "/home/ubuntu/video-recommendation/runs/1664306137-elite-chameleon"
2022/09/28 00:38:57 DEBUG : Creating backend with remote "r2:/checkpoints/elite-chameleon"
2022/09/28 00:38:57 DEBUG : fs cache: renaming cache item "r2:/checkpoints/elite-chameleon" to be canonical "r2:checkpoints/elite-chameleon"
2022-09-28 00:38:57 NOTICE: checkpoints/latest-rank0.pt: Can't follow symlink without -L/--copy-links
2022-09-28 00:38:57 DEBUG : logs-rank1.txt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:38:57 DEBUG : logs-rank1.txt: Unchanged skipping
2022-09-28 00:38:58 DEBUG : S3 bucket checkpoints path elite-chameleon: Waiting for checks to finish
2022-09-28 00:38:58 DEBUG : checkpoints/ep0-ba1035-rank0.pt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:38:58 DEBUG : checkpoints/ep0-ba1035-rank0.pt: Unchanged skipping
2022-09-28 00:38:58 DEBUG : logs-rank2.txt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:38:58 DEBUG : logs-rank2.txt: Unchanged skipping
2022-09-28 00:38:58 DEBUG : logs-rank3.txt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:38:58 DEBUG : logs-rank3.txt: Unchanged skipping
2022-09-28 00:38:58 DEBUG : logs-rank4.txt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:38:58 DEBUG : logs-rank4.txt: Unchanged skipping
2022-09-28 00:38:58 DEBUG : logs-rank6.txt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:38:58 DEBUG : logs-rank6.txt: Unchanged skipping
2022-09-28 00:38:58 DEBUG : logs-rank5.txt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:38:58 DEBUG : logs-rank5.txt: Unchanged skipping
2022-09-28 00:38:58 DEBUG : logs-rank0.txt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:38:58 DEBUG : logs-rank0.txt: Unchanged skipping
2022-09-28 00:38:58 DEBUG : checkpoints/ep0-ba115-rank0.pt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:38:58 DEBUG : checkpoints/ep0-ba115-rank0.pt: Unchanged skipping
2022-09-28 00:38:58 DEBUG : logs-rank7.txt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:38:58 DEBUG : logs-rank7.txt: Unchanged skipping
2022-09-28 00:38:58 DEBUG : checkpoints/ep0-ba1150-rank0.pt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:38:58 DEBUG : checkpoints/ep0-ba1150-rank0.pt: Unchanged skipping
2022-09-28 00:38:58 DEBUG : checkpoints/ep0-ba1265-rank0.pt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:38:58 DEBUG : checkpoints/ep0-ba1265-rank0.pt: Unchanged skipping
2022-09-28 00:38:58 DEBUG : checkpoints/ep0-ba575-rank0.pt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:38:58 DEBUG : checkpoints/ep0-ba575-rank0.pt: Unchanged skipping
2022-09-28 00:38:58 DEBUG : checkpoints/ep0-ba1380-rank0.pt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:38:58 DEBUG : checkpoints/ep0-ba1380-rank0.pt: Unchanged skipping
2022-09-28 00:38:58 DEBUG : checkpoints/ep0-ba345-rank0.pt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:38:58 DEBUG : checkpoints/ep0-ba345-rank0.pt: Unchanged skipping
2022-09-28 00:38:58 DEBUG : checkpoints/ep0-ba230-rank0.pt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:38:58 DEBUG : checkpoints/ep0-ba230-rank0.pt: Unchanged skipping
2022-09-28 00:38:58 DEBUG : checkpoints/ep0-ba460-rank0.pt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:38:58 DEBUG : checkpoints/ep0-ba460-rank0.pt: Unchanged skipping
2022-09-28 00:38:58 DEBUG : checkpoints/ep0-ba690-rank0.pt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:38:58 DEBUG : checkpoints/ep0-ba690-rank0.pt: Unchanged skipping
2022-09-28 00:38:58 DEBUG : checkpoints/ep0-ba805-rank0.pt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:38:58 DEBUG : checkpoints/ep0-ba805-rank0.pt: Unchanged skipping
2022-09-28 00:38:58 DEBUG : S3 bucket checkpoints path elite-chameleon: Waiting for transfers to finish
2022-09-28 00:38:59 DEBUG : checkpoints/ep0-ba920-rank0.pt: multipart upload starting chunk 1 size 5Mi offset 0/243.288Mi
2022-09-28 00:38:59 DEBUG : checkpoints/ep0-ba920-rank0.pt: multipart upload starting chunk 2 size 5Mi offset 5Mi/243.288Mi
2022-09-28 00:38:59 DEBUG : checkpoints/ep0-ba920-rank0.pt: multipart upload starting chunk 3 size 5Mi offset 10Mi/243.288Mi
2022-09-28 00:38:59 DEBUG : checkpoints/ep0-ba920-rank0.pt: multipart upload starting chunk 4 size 5Mi offset 15Mi/243.288Mi
2022-09-28 00:38:59 DEBUG : checkpoints/ep0-ba920-rank0.pt: multipart upload starting chunk 5 size 5Mi offset 20Mi/243.288Mi
2022-09-28 00:39:00 DEBUG : checkpoints/ep0-ba920-rank0.pt: multipart upload starting chunk 6 size 5Mi offset 25Mi/243.288Mi
2022-09-28 00:39:00 DEBUG : checkpoints/ep0-ba920-rank0.pt: multipart upload starting chunk 7 size 5Mi offset 30Mi/243.288Mi
2022-09-28 00:39:00 DEBUG : checkpoints/ep0-ba920-rank0.pt: multipart upload starting chunk 8 size 5Mi offset 35Mi/243.288Mi
2022-09-28 00:39:01 DEBUG : pacer: low level retry 1/2 (error RequestCanceled: request context canceled
caused by: context canceled)
2022-09-28 00:39:01 DEBUG : pacer: low level retry 1/2 (error RequestCanceled: request context canceled
caused by: context canceled)
2022-09-28 00:39:01 DEBUG : pacer: Rate limited, increasing sleep to 10ms
2022-09-28 00:39:01 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2022-09-28 00:39:01 DEBUG : pacer: low level retry 1/2 (error RequestCanceled: request context canceled
caused by: context canceled)
2022-09-28 00:39:01 DEBUG : pacer: Rate limited, increasing sleep to 40ms
2022-09-28 00:39:01 DEBUG : pacer: low level retry 2/2 (error RequestCanceled: request context canceled
caused by: context canceled)
2022-09-28 00:39:01 DEBUG : pacer: Rate limited, increasing sleep to 80ms
2022-09-28 00:39:01 DEBUG : pacer: low level retry 2/2 (error RequestCanceled: request context canceled
caused by: context canceled)
2022-09-28 00:39:01 DEBUG : pacer: Rate limited, increasing sleep to 160ms
2022-09-28 00:39:01 DEBUG : pacer: low level retry 2/2 (error RequestCanceled: request context canceled
caused by: context canceled)
2022-09-28 00:39:01 DEBUG : pacer: Rate limited, increasing sleep to 320ms
2022-09-28 00:39:01 DEBUG : checkpoints/ep0-ba920-rank0.pt: Cancelling multipart upload
2022-09-28 00:39:02 DEBUG : pacer: Reducing sleep to 240ms
2022-09-28 00:39:02 ERROR : checkpoints/ep0-ba920-rank0.pt: Failed to copy: multipart upload failed to upload part: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method. 
        status code: 403, request id: , host id:
2022-09-28 00:39:02 ERROR : Attempt 1/3 failed with 1 errors and: multipart upload failed to upload part: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method. 
        status code: 403, request id: , host id:
2022-09-28 00:39:02 DEBUG : pacer: Reducing sleep to 180ms
2022-09-28 00:39:02 NOTICE: checkpoints/latest-rank0.pt: Can't follow symlink without -L/--copy-links
2022-09-28 00:39:02 DEBUG : pacer: Reducing sleep to 135ms
2022-09-28 00:39:02 DEBUG : S3 bucket checkpoints path elite-chameleon: Waiting for checks to finish
2022-09-28 00:39:03 DEBUG : pacer: Reducing sleep to 101.25ms
2022-09-28 00:39:03 DEBUG : logs-rank2.txt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:39:03 DEBUG : logs-rank2.txt: Unchanged skipping
2022-09-28 00:39:03 DEBUG : pacer: Reducing sleep to 75.9375ms
2022-09-28 00:39:03 DEBUG : logs-rank4.txt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:39:03 DEBUG : logs-rank4.txt: Unchanged skipping
2022-09-28 00:39:03 DEBUG : pacer: Reducing sleep to 56.953125ms
2022-09-28 00:39:03 DEBUG : logs-rank5.txt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:39:03 DEBUG : logs-rank5.txt: Unchanged skipping
2022-09-28 00:39:03 DEBUG : pacer: Reducing sleep to 42.714843ms
2022-09-28 00:39:03 DEBUG : logs-rank0.txt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:39:03 DEBUG : logs-rank0.txt: Unchanged skipping
2022-09-28 00:39:03 DEBUG : pacer: Reducing sleep to 32.036132ms
2022-09-28 00:39:03 DEBUG : logs-rank1.txt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:39:03 DEBUG : logs-rank1.txt: Unchanged skipping
2022-09-28 00:39:03 DEBUG : pacer: Reducing sleep to 24.027099ms
2022-09-28 00:39:03 DEBUG : logs-rank6.txt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:39:03 DEBUG : logs-rank6.txt: Unchanged skipping
2022-09-28 00:39:03 DEBUG : pacer: Reducing sleep to 18.020324ms
2022-09-28 00:39:03 DEBUG : logs-rank3.txt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:39:03 DEBUG : logs-rank3.txt: Unchanged skipping
2022-09-28 00:39:03 DEBUG : pacer: Reducing sleep to 13.515243ms
2022-09-28 00:39:03 DEBUG : logs-rank7.txt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:39:03 DEBUG : logs-rank7.txt: Unchanged skipping
2022-09-28 00:39:03 DEBUG : pacer: Reducing sleep to 10.136432ms
2022-09-28 00:39:03 DEBUG : checkpoints/ep0-ba1035-rank0.pt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:39:03 DEBUG : checkpoints/ep0-ba1035-rank0.pt: Unchanged skipping
2022-09-28 00:39:03 DEBUG : pacer: Reducing sleep to 0s
2022-09-28 00:39:03 DEBUG : checkpoints/ep0-ba115-rank0.pt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:39:03 DEBUG : checkpoints/ep0-ba115-rank0.pt: Unchanged skipping
2022-09-28 00:39:03 DEBUG : checkpoints/ep0-ba1150-rank0.pt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:39:03 DEBUG : checkpoints/ep0-ba1150-rank0.pt: Unchanged skipping
2022-09-28 00:39:03 DEBUG : checkpoints/ep0-ba1380-rank0.pt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:39:03 DEBUG : checkpoints/ep0-ba1380-rank0.pt: Unchanged skipping
2022-09-28 00:39:03 DEBUG : checkpoints/ep0-ba1265-rank0.pt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:39:03 DEBUG : checkpoints/ep0-ba1265-rank0.pt: Unchanged skipping
2022-09-28 00:39:03 DEBUG : checkpoints/ep0-ba230-rank0.pt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:39:03 DEBUG : checkpoints/ep0-ba230-rank0.pt: Unchanged skipping
2022-09-28 00:39:03 DEBUG : checkpoints/ep0-ba460-rank0.pt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:39:03 DEBUG : checkpoints/ep0-ba460-rank0.pt: Unchanged skipping
2022-09-28 00:39:03 DEBUG : checkpoints/ep0-ba805-rank0.pt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:39:03 DEBUG : checkpoints/ep0-ba805-rank0.pt: Unchanged skipping
2022-09-28 00:39:03 DEBUG : checkpoints/ep0-ba345-rank0.pt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:39:03 DEBUG : checkpoints/ep0-ba345-rank0.pt: Unchanged skipping
2022-09-28 00:39:03 DEBUG : checkpoints/ep0-ba690-rank0.pt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:39:03 DEBUG : checkpoints/ep0-ba690-rank0.pt: Unchanged skipping
2022-09-28 00:39:03 DEBUG : checkpoints/ep0-ba575-rank0.pt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:39:03 DEBUG : checkpoints/ep0-ba575-rank0.pt: Unchanged skipping
2022-09-28 00:39:03 DEBUG : S3 bucket checkpoints path elite-chameleon: Waiting for transfers to finish
2022-09-28 00:39:04 DEBUG : checkpoints/ep0-ba920-rank0.pt: multipart upload starting chunk 1 size 5Mi offset 0/243.288Mi
2022-09-28 00:39:04 DEBUG : checkpoints/ep0-ba920-rank0.pt: multipart upload starting chunk 2 size 5Mi offset 5Mi/243.288Mi
2022-09-28 00:39:04 DEBUG : checkpoints/ep0-ba920-rank0.pt: multipart upload starting chunk 3 size 5Mi offset 10Mi/243.288Mi
2022-09-28 00:39:04 DEBUG : checkpoints/ep0-ba920-rank0.pt: multipart upload starting chunk 4 size 5Mi offset 15Mi/243.288Mi
2022-09-28 00:39:04 DEBUG : checkpoints/ep0-ba920-rank0.pt: multipart upload starting chunk 5 size 5Mi offset 20Mi/243.288Mi
2022-09-28 00:39:05 DEBUG : checkpoints/ep0-ba920-rank0.pt: multipart upload starting chunk 6 size 5Mi offset 25Mi/243.288Mi
2022-09-28 00:39:05 DEBUG : checkpoints/ep0-ba920-rank0.pt: multipart upload starting chunk 7 size 5Mi offset 30Mi/243.288Mi
2022-09-28 00:39:06 DEBUG : checkpoints/ep0-ba920-rank0.pt: multipart upload starting chunk 8 size 5Mi offset 35Mi/243.288Mi
2022-09-28 00:39:06 DEBUG : checkpoints/ep0-ba920-rank0.pt: multipart upload starting chunk 9 size 5Mi offset 40Mi/243.288Mi
2022-09-28 00:39:06 DEBUG : pacer: low level retry 1/2 (error RequestCanceled: request context canceled
caused by: context canceled)
2022-09-28 00:39:06 DEBUG : pacer: Rate limited, increasing sleep to 10ms
2022-09-28 00:39:06 DEBUG : pacer: low level retry 1/2 (error RequestCanceled: request context canceled
caused by: context canceled)
2022-09-28 00:39:06 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2022-09-28 00:39:06 DEBUG : pacer: low level retry 1/2 (error RequestCanceled: request context canceled
caused by: context canceled)
2022-09-28 00:39:06 DEBUG : pacer: Rate limited, increasing sleep to 40ms
2022-09-28 00:39:06 DEBUG : pacer: low level retry 2/2 (error RequestCanceled: request context canceled
caused by: context canceled)
2022-09-28 00:39:06 DEBUG : pacer: Rate limited, increasing sleep to 80ms
2022-09-28 00:39:06 DEBUG : pacer: low level retry 2/2 (error RequestCanceled: request context canceled
caused by: context canceled)
2022-09-28 00:39:06 DEBUG : pacer: Rate limited, increasing sleep to 160ms
2022-09-28 00:39:06 DEBUG : pacer: low level retry 2/2 (error RequestCanceled: request context canceled
caused by: context canceled)
2022-09-28 00:39:06 DEBUG : pacer: Rate limited, increasing sleep to 320ms
2022-09-28 00:39:06 DEBUG : checkpoints/ep0-ba920-rank0.pt: Cancelling multipart upload
2022-09-28 00:39:07 DEBUG : pacer: Reducing sleep to 240ms
2022-09-28 00:39:07 ERROR : checkpoints/ep0-ba920-rank0.pt: Failed to copy: multipart upload failed to upload part: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method. 
        status code: 403, request id: , host id:
2022-09-28 00:39:07 ERROR : Attempt 2/3 failed with 1 errors and: multipart upload failed to upload part: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method. 
        status code: 403, request id: , host id:
2022-09-28 00:39:07 DEBUG : pacer: Reducing sleep to 180ms
2022-09-28 00:39:07 NOTICE: checkpoints/latest-rank0.pt: Can't follow symlink without -L/--copy-links
2022-09-28 00:39:07 DEBUG : pacer: Reducing sleep to 135ms
2022-09-28 00:39:07 DEBUG : S3 bucket checkpoints path elite-chameleon: Waiting for checks to finish
2022-09-28 00:39:07 DEBUG : pacer: Reducing sleep to 101.25ms
2022-09-28 00:39:07 DEBUG : logs-rank1.txt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:39:07 DEBUG : logs-rank1.txt: Unchanged skipping
2022-09-28 00:39:07 DEBUG : pacer: Reducing sleep to 75.9375ms
2022-09-28 00:39:07 DEBUG : logs-rank2.txt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:39:07 DEBUG : logs-rank2.txt: Unchanged skipping
2022-09-28 00:39:08 DEBUG : pacer: Reducing sleep to 56.953125ms
2022-09-28 00:39:08 DEBUG : logs-rank0.txt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:39:08 DEBUG : logs-rank0.txt: Unchanged skipping
2022-09-28 00:39:08 DEBUG : pacer: Reducing sleep to 42.714843ms
2022-09-28 00:39:08 DEBUG : logs-rank3.txt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:39:08 DEBUG : logs-rank3.txt: Unchanged skipping
2022-09-28 00:39:08 DEBUG : pacer: Reducing sleep to 32.036132ms
2022-09-28 00:39:08 DEBUG : logs-rank5.txt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:39:08 DEBUG : logs-rank5.txt: Unchanged skipping
2022-09-28 00:39:08 DEBUG : pacer: Reducing sleep to 24.027099ms
2022-09-28 00:39:08 DEBUG : logs-rank6.txt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:39:08 DEBUG : logs-rank6.txt: Unchanged skipping
2022-09-28 00:39:08 DEBUG : pacer: Reducing sleep to 18.020324ms
2022-09-28 00:39:08 DEBUG : logs-rank7.txt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:39:08 DEBUG : logs-rank7.txt: Unchanged skipping
2022-09-28 00:39:08 DEBUG : pacer: Reducing sleep to 13.515243ms
2022-09-28 00:39:08 DEBUG : logs-rank4.txt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:39:08 DEBUG : logs-rank4.txt: Unchanged skipping
2022-09-28 00:39:08 DEBUG : pacer: Reducing sleep to 10.136432ms
2022-09-28 00:39:08 DEBUG : checkpoints/ep0-ba1035-rank0.pt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:39:08 DEBUG : checkpoints/ep0-ba1035-rank0.pt: Unchanged skipping
2022-09-28 00:39:08 DEBUG : pacer: Reducing sleep to 0s
2022-09-28 00:39:08 DEBUG : checkpoints/ep0-ba115-rank0.pt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:39:08 DEBUG : checkpoints/ep0-ba115-rank0.pt: Unchanged skipping
2022-09-28 00:39:08 DEBUG : checkpoints/ep0-ba1150-rank0.pt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:39:08 DEBUG : checkpoints/ep0-ba1150-rank0.pt: Unchanged skipping
2022-09-28 00:39:08 DEBUG : checkpoints/ep0-ba1380-rank0.pt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:39:08 DEBUG : checkpoints/ep0-ba1380-rank0.pt: Unchanged skipping
2022-09-28 00:39:08 DEBUG : checkpoints/ep0-ba805-rank0.pt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:39:08 DEBUG : checkpoints/ep0-ba805-rank0.pt: Unchanged skipping
2022-09-28 00:39:08 DEBUG : checkpoints/ep0-ba1265-rank0.pt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:39:08 DEBUG : checkpoints/ep0-ba1265-rank0.pt: Unchanged skipping
2022-09-28 00:39:08 DEBUG : checkpoints/ep0-ba345-rank0.pt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:39:08 DEBUG : checkpoints/ep0-ba345-rank0.pt: Unchanged skipping
2022-09-28 00:39:08 DEBUG : checkpoints/ep0-ba230-rank0.pt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:39:08 DEBUG : checkpoints/ep0-ba230-rank0.pt: Unchanged skipping
2022-09-28 00:39:08 DEBUG : checkpoints/ep0-ba460-rank0.pt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:39:08 DEBUG : checkpoints/ep0-ba460-rank0.pt: Unchanged skipping
2022-09-28 00:39:08 DEBUG : checkpoints/ep0-ba575-rank0.pt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:39:08 DEBUG : checkpoints/ep0-ba575-rank0.pt: Unchanged skipping
2022-09-28 00:39:08 DEBUG : checkpoints/ep0-ba690-rank0.pt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-09-28 00:39:08 DEBUG : checkpoints/ep0-ba690-rank0.pt: Unchanged skipping
2022-09-28 00:39:08 DEBUG : S3 bucket checkpoints path elite-chameleon: Waiting for transfers to finish
2022-09-28 00:39:08 DEBUG : checkpoints/ep0-ba920-rank0.pt: multipart upload starting chunk 1 size 5Mi offset 0/243.288Mi
2022-09-28 00:39:08 DEBUG : checkpoints/ep0-ba920-rank0.pt: multipart upload starting chunk 2 size 5Mi offset 5Mi/243.288Mi
2022-09-28 00:39:08 DEBUG : checkpoints/ep0-ba920-rank0.pt: multipart upload starting chunk 3 size 5Mi offset 10Mi/243.288Mi
2022-09-28 00:39:08 DEBUG : checkpoints/ep0-ba920-rank0.pt: multipart upload starting chunk 4 size 5Mi offset 15Mi/243.288Mi
2022-09-28 00:39:09 DEBUG : checkpoints/ep0-ba920-rank0.pt: multipart upload starting chunk 5 size 5Mi offset 20Mi/243.288Mi
2022-09-28 00:39:10 DEBUG : checkpoints/ep0-ba920-rank0.pt: multipart upload starting chunk 6 size 5Mi offset 25Mi/243.288Mi
2022-09-28 00:39:10 DEBUG : checkpoints/ep0-ba920-rank0.pt: multipart upload starting chunk 7 size 5Mi offset 30Mi/243.288Mi
2022-09-28 00:39:11 DEBUG : checkpoints/ep0-ba920-rank0.pt: multipart upload starting chunk 8 size 5Mi offset 35Mi/243.288Mi
2022-09-28 00:39:11 DEBUG : checkpoints/ep0-ba920-rank0.pt: multipart upload starting chunk 9 size 5Mi offset 40Mi/243.288Mi
2022-09-28 00:39:12 DEBUG : checkpoints/ep0-ba920-rank0.pt: multipart upload starting chunk 10 size 5Mi offset 45Mi/243.288Mi
2022-09-28 00:39:12 DEBUG : pacer: low level retry 1/2 (error RequestCanceled: request context canceled
caused by: context canceled)
2022-09-28 00:39:12 DEBUG : pacer: Rate limited, increasing sleep to 10ms
2022-09-28 00:39:12 DEBUG : pacer: low level retry 1/2 (error RequestCanceled: request context canceled
caused by: context canceled)
2022-09-28 00:39:12 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2022-09-28 00:39:12 DEBUG : pacer: low level retry 2/2 (error RequestCanceled: request context canceled
caused by: context canceled)
2022-09-28 00:39:12 DEBUG : pacer: Rate limited, increasing sleep to 40ms
2022-09-28 00:39:12 DEBUG : pacer: low level retry 2/2 (error RequestCanceled: request context canceled
caused by: context canceled)
2022-09-28 00:39:12 DEBUG : pacer: Rate limited, increasing sleep to 80ms
2022-09-28 00:39:12 DEBUG : pacer: low level retry 1/2 (error RequestCanceled: request context canceled
caused by: context canceled)
2022-09-28 00:39:12 DEBUG : pacer: Rate limited, increasing sleep to 160ms
2022-09-28 00:39:12 DEBUG : pacer: low level retry 2/2 (error RequestCanceled: request context canceled
caused by: context canceled)
2022-09-28 00:39:12 DEBUG : pacer: Rate limited, increasing sleep to 320ms
2022-09-28 00:39:12 DEBUG : checkpoints/ep0-ba920-rank0.pt: Cancelling multipart upload
2022-09-28 00:39:12 DEBUG : pacer: Reducing sleep to 240ms
2022-09-28 00:39:12 ERROR : checkpoints/ep0-ba920-rank0.pt: Failed to copy: multipart upload failed to upload part: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method. 
        status code: 403, request id: , host id:
2022-09-28 00:39:12 ERROR : Attempt 3/3 failed with 1 errors and: multipart upload failed to upload part: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method. 
        status code: 403, request id: , host id:
Transferred:          135 MiB / 135 MiB, 100%, 9.287 MiB/s, ETA 0s
Errors:                 1 (retrying may help)
Checks:                57 / 57, 100%
Elapsed time:        15.4s
2022/09/28 00:39:12 INFO  : 
Transferred:   	      135 MiB / 135 MiB, 100%, 9.287 MiB/s, ETA 0s
Errors:                 1 (retrying may help)
Checks:                57 / 57, 100%
Elapsed time:        15.4s

2022/09/28 00:39:12 DEBUG : 20 go routines active
2022/09/28 00:39:12 Failed to copy: multipart upload failed to upload part: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method. 
	status code: 403, request id: , host id:

This does not seem to happen for smaller files.

2

hi,

not sure what the exact issue is but your remote does not seem to match the example remote in rclone docs.
for example,
in your remote, provider = Other and in the rclone example, provider = Cloudflare

and these are missing
region = auto
and
acl = private

3

@asdffdsa Thanks for the response. I updated my rclone config file to match the docs, but the same errors are still occurring.

Specifically, I used:

  • provider = Cloudflare
  • acl = private
4

So this problem is intermittent?

I tried to upload some files of about 250M to R2 and they all worked fine..

Can you just try to copy that one file with a bit more logging

rclone copy runs/1664306137-elite-chameleon/checkpoints/ep0-ba920-rank0.pt r2:/checkpoints/elite-chameleon/ -vv --retries 1 --low-level-retries 1 --dump responses 2>&1 | tee rclone.log

And attach rclone.log here (or put on pastbin/github gist/wherever if too big).

Would it be possible for me to have a copy of that file so I can try the upload from here? If so please share with nick@craig-wood.com somehow!

I suspect this is teething problems at R2 - they tend to come and go quite quickly.

5

I ran the command:

head -c 1G </dev/urandom > tempfile
rclone copy tempfile r2:/temp2 -vv --retries 1 --low-level-retries 1 --dump responses 2>&1 | tee rclone.log

Here's the output:

2022/09/28 21:47:06 DEBUG : rclone: Version "v1.59.2" starting with parameters ["rclone" "copy" "tempfile" "r2:/temp2" "-vv" "--retries" "1" "--low-level-retries" "1" "--dump" "responses"]
2022/09/28 21:47:06 DEBUG : Creating backend with remote "tempfile"
2022/09/28 21:47:06 DEBUG : Using config file from "/home/ubuntu/.config/rclone/rclone.conf"
2022/09/28 21:47:06 DEBUG : fs cache: adding new entry for parent of "tempfile", "/home/ubuntu"
2022/09/28 21:47:06 DEBUG : Creating backend with remote "r2:/temp2"
2022/09/28 21:47:06 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2022/09/28 21:47:06 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2022/09/28 21:47:06 DEBUG : fs cache: renaming cache item "r2:/temp2" to be canonical "r2:temp2"
2022/09/28 21:47:06 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/09/28 21:47:06 DEBUG : HTTP REQUEST (req 0xc000e16100)
2022/09/28 21:47:06 DEBUG : HEAD /temp2/tempfile HTTP/1.1
Host: 2d0e76e05bb785204f4f720a65f6281b.r2.cloudflarestorage.com
User-Agent: rclone/v1.59.2
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220928T214706Z

2022/09/28 21:47:06 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/09/28 21:47:06 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/09/28 21:47:06 DEBUG : HTTP RESPONSE (req 0xc000e16100)
2022/09/28 21:47:06 DEBUG : HTTP/1.1 404 Not Found
Connection: close
Cf-Ray: 751facf2ecf181c4-IAD
Connection: keep-alive
Content-Type: text/plain;charset=UTF-8
Date: Wed, 28 Sep 2022 21:47:06 GMT
Server: cloudflare

2022/09/28 21:47:06 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/09/28 21:47:06 DEBUG : tempfile: Need to transfer - File not found at Destination
2022/09/28 21:47:06 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/09/28 21:47:06 DEBUG : HTTP REQUEST (req 0xc000e16500)
2022/09/28 21:47:06 DEBUG : PUT /temp2 HTTP/1.1
Host: 2d0e76e05bb785204f4f720a65f6281b.r2.cloudflarestorage.com
User-Agent: rclone/v1.59.2
Content-Length: 0
Authorization: XXXX
X-Amz-Acl: private
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220928T214706Z
Accept-Encoding: gzip

2022/09/28 21:47:06 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/09/28 21:47:06 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/09/28 21:47:06 DEBUG : HTTP RESPONSE (req 0xc000e16500)
2022/09/28 21:47:06 DEBUG : HTTP/1.1 409 Conflict
Content-Length: 170
Cf-Ray: 751facf539a081c4-IAD
Connection: keep-alive
Content-Type: application/xml
Date: Wed, 28 Sep 2022 21:47:06 GMT
Server: cloudflare
Vary: Accept-Encoding

<?xml version="1.0" encoding="UTF-8"?><Error><Code>BucketAlreadyOwnedByYou</Code><Message>The bucket you tried to create already exists, and you own it.</Message></Error>
2022/09/28 21:47:06 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/09/28 21:47:08 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/09/28 21:47:08 DEBUG : HTTP REQUEST (req 0xc000b84600)
2022/09/28 21:47:08 DEBUG : POST /temp2/tempfile?uploads= HTTP/1.1
Host: 2d0e76e05bb785204f4f720a65f6281b.r2.cloudflarestorage.com
User-Agent: rclone/v1.59.2
Content-Length: 0
Authorization: XXXX
Content-Type: application/octet-stream
X-Amz-Acl: private
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220928T214708Z
X-Amz-Meta-Md5chksum: OKSw8+m61RJfBHupuQMSdQ==
X-Amz-Meta-Mtime: 1664400618.606433775
Accept-Encoding: gzip

2022/09/28 21:47:08 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/09/28 21:47:08 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/09/28 21:47:08 DEBUG : HTTP RESPONSE (req 0xc000b84600)
2022/09/28 21:47:08 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Cf-Ray: 751fad035b5381c4-IAD
Connection: keep-alive
Content-Type: application/xml
Date: Wed, 28 Sep 2022 21:47:08 GMT
Server: cloudflare
Vary: Accept-Encoding

217
<?xml version="1.0" encoding="UTF-8"?><InitiateMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Bucket>temp2</Bucket><Key>tempfile</Key><UploadId>ABA5HzhZKiobirEUckHtYFA8cgWGX/LPGS8IL/EringaHZRqWsOD+NIkLLmM53q8Ech65eZktRV/SzWOQTEOxqA++5xNur30TsymFgav3XUb4yLJ+6ElpLlhoNTUVc6XAF/+Yzfqw6NIQYoWNq3eQloOg/ifSyRavOmuQVL8fnT51VQSkXdtFgFsWmKqr+2xIH9oLKj0NCi3cFXyXeG10tmHUOzyUnK2PBZ7qG7wTpV9/n13oi4xHf8Xt2Ba5A/gM5j3IpV9oKBCoyp4xmJavIW6uEPCpFH2at5mL5OFOqyQPlFfEmYMolfyAFYcYDGdxg==</UploadId></InitiateMultipartUploadResult>
0

2022/09/28 21:47:08 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/09/28 21:47:08 DEBUG : tempfile: multipart upload starting chunk 1 size 5Mi offset 0/1Gi
2022/09/28 21:47:08 DEBUG : tempfile: multipart upload starting chunk 2 size 5Mi offset 5Mi/1Gi
2022/09/28 21:47:08 DEBUG : tempfile: multipart upload starting chunk 3 size 5Mi offset 10Mi/1Gi
2022/09/28 21:47:08 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/09/28 21:47:08 DEBUG : HTTP REQUEST (req 0xc000e16100)
2022/09/28 21:47:08 DEBUG : PUT /temp2/tempfile?partNumber=1&uploadId=ABA5HzhZKiobirEUckHtYFA8cgWGX%2FLPGS8IL%2FEringaHZRqWsOD%2BNIkLLmM53q8Ech65eZktRV%2FSzWOQTEOxqA%2B%2B5xNur30TsymFgav3XUb4yLJ%2B6ElpLlhoNTUVc6XAF%2F%2BYzfqw6NIQYoWNq3eQloOg%2FifSyRavOmuQVL8fnT51VQSkXdtFgFsWmKqr%2B2xIH9oLKj0NCi3cFXyXeG10tmHUOzyUnK2PBZ7qG7wTpV9%2Fn13oi4xHf8Xt2Ba5A%2FgM5j3IpV9oKBCoyp4xmJavIW6uEPCpFH2at5mL5OFOqyQPlFfEmYMolfyAFYcYDGdxg%3D%3D HTTP/1.1
Host: 2d0e76e05bb785204f4f720a65f6281b.r2.cloudflarestorage.com
User-Agent: rclone/v1.59.2
Content-Length: 5242880
Authorization: XXXX
Content-Md5: f5yTj4dGUp7pLd3xHivBVw==
Expect: 100-Continue
X-Amz-Content-Sha256: 08995cbbc026295df3dc28ca8ee90568a114d80d8273796be0faf376b1898592
X-Amz-Date: 20220928T214708Z
Accept-Encoding: gzip

2022/09/28 21:47:08 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/09/28 21:47:08 DEBUG : tempfile: multipart upload starting chunk 4 size 5Mi offset 15Mi/1Gi
2022/09/28 21:47:08 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/09/28 21:47:08 DEBUG : HTTP REQUEST (req 0xc000b84600)
2022/09/28 21:47:08 DEBUG : PUT /temp2/tempfile?partNumber=2&uploadId=ABA5HzhZKiobirEUckHtYFA8cgWGX%2FLPGS8IL%2FEringaHZRqWsOD%2BNIkLLmM53q8Ech65eZktRV%2FSzWOQTEOxqA%2B%2B5xNur30TsymFgav3XUb4yLJ%2B6ElpLlhoNTUVc6XAF%2F%2BYzfqw6NIQYoWNq3eQloOg%2FifSyRavOmuQVL8fnT51VQSkXdtFgFsWmKqr%2B2xIH9oLKj0NCi3cFXyXeG10tmHUOzyUnK2PBZ7qG7wTpV9%2Fn13oi4xHf8Xt2Ba5A%2FgM5j3IpV9oKBCoyp4xmJavIW6uEPCpFH2at5mL5OFOqyQPlFfEmYMolfyAFYcYDGdxg%3D%3D HTTP/1.1
Host: 2d0e76e05bb785204f4f720a65f6281b.r2.cloudflarestorage.com
User-Agent: rclone/v1.59.2
Content-Length: 5242880
Authorization: XXXX
Content-Md5: rVE7WDKzXzcPz3fE4Cke4w==
Expect: 100-Continue
X-Amz-Content-Sha256: 449fa97d0b26b3457e6a845c22fc4f7b70fb338d4320c8f951c5ca34e898ab89
X-Amz-Date: 20220928T214708Z
Accept-Encoding: gzip

2022/09/28 21:47:08 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/09/28 21:47:08 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/09/28 21:47:08 DEBUG : HTTP REQUEST (req 0xc00027e500)
2022/09/28 21:47:08 DEBUG : PUT /temp2/tempfile?partNumber=3&uploadId=ABA5HzhZKiobirEUckHtYFA8cgWGX%2FLPGS8IL%2FEringaHZRqWsOD%2BNIkLLmM53q8Ech65eZktRV%2FSzWOQTEOxqA%2B%2B5xNur30TsymFgav3XUb4yLJ%2B6ElpLlhoNTUVc6XAF%2F%2BYzfqw6NIQYoWNq3eQloOg%2FifSyRavOmuQVL8fnT51VQSkXdtFgFsWmKqr%2B2xIH9oLKj0NCi3cFXyXeG10tmHUOzyUnK2PBZ7qG7wTpV9%2Fn13oi4xHf8Xt2Ba5A%2FgM5j3IpV9oKBCoyp4xmJavIW6uEPCpFH2at5mL5OFOqyQPlFfEmYMolfyAFYcYDGdxg%3D%3D HTTP/1.1
Host: 2d0e76e05bb785204f4f720a65f6281b.r2.cloudflarestorage.com
User-Agent: rclone/v1.59.2
Content-Length: 5242880
Authorization: XXXX
Content-Md5: IUiN34WF3povDsgBjSUnrg==
Expect: 100-Continue
X-Amz-Content-Sha256: 9a17e2daf71bb2c25138355afa9c544b28a1989e4c97ad98741286f14b206161
X-Amz-Date: 20220928T214708Z
Accept-Encoding: gzip

2022/09/28 21:47:08 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/09/28 21:47:08 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/09/28 21:47:08 DEBUG : HTTP REQUEST (req 0xc000d28a00)
2022/09/28 21:47:08 DEBUG : PUT /temp2/tempfile?partNumber=4&uploadId=ABA5HzhZKiobirEUckHtYFA8cgWGX%2FLPGS8IL%2FEringaHZRqWsOD%2BNIkLLmM53q8Ech65eZktRV%2FSzWOQTEOxqA%2B%2B5xNur30TsymFgav3XUb4yLJ%2B6ElpLlhoNTUVc6XAF%2F%2BYzfqw6NIQYoWNq3eQloOg%2FifSyRavOmuQVL8fnT51VQSkXdtFgFsWmKqr%2B2xIH9oLKj0NCi3cFXyXeG10tmHUOzyUnK2PBZ7qG7wTpV9%2Fn13oi4xHf8Xt2Ba5A%2FgM5j3IpV9oKBCoyp4xmJavIW6uEPCpFH2at5mL5OFOqyQPlFfEmYMolfyAFYcYDGdxg%3D%3D HTTP/1.1
Host: 2d0e76e05bb785204f4f720a65f6281b.r2.cloudflarestorage.com
User-Agent: rclone/v1.59.2
Content-Length: 5242880
Authorization: XXXX
Content-Md5: NJiZiyzxSOV6o2Xt85GGFQ==
Expect: 100-Continue
X-Amz-Content-Sha256: e9e003b21cee82a6cfe1cf936db99b8603b4c7685a5916c0ee35145dc5b43d10
X-Amz-Date: 20220928T214708Z
Accept-Encoding: gzip

2022/09/28 21:47:08 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/09/28 21:47:09 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/09/28 21:47:09 DEBUG : HTTP RESPONSE (req 0xc000e16100)
2022/09/28 21:47:09 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Cf-Ray: 751fad04ce6a81c4-IAD
Connection: keep-alive
Date: Wed, 28 Sep 2022 21:47:09 GMT
Etag: "AENzW/jOTPXDJ0qBfBVrdFG69JZPRlesV52JXtJK2Cfmyb99bZ7WE2g7GYtqqtf9reQYjK2IgJGC7KfWpk0IP/JCAFD7/HVi3rLk1V0viBP0v3yg6lmVADaUKcrNFLidKwbtxmPThftZ5wyx8YeD62Hcc9UVSXab/sd7hyTDULC2QYht7HC2PJPiWQY5Y431TUpyXfvAJndH6H3IIuH9uW61v2rDnpKQFcNeiidKbQmrySE7s3du2MjHESE0C/gT8Q=="
Server: cloudflare
Vary: Accept-Encoding

2022/09/28 21:47:09 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/09/28 21:47:09 DEBUG : tempfile: multipart upload starting chunk 5 size 5Mi offset 20Mi/1Gi
2022/09/28 21:47:09 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/09/28 21:47:09 DEBUG : HTTP REQUEST (req 0xc0005b9500)
2022/09/28 21:47:09 DEBUG : PUT /temp2/tempfile?partNumber=5&uploadId=ABA5HzhZKiobirEUckHtYFA8cgWGX%2FLPGS8IL%2FEringaHZRqWsOD%2BNIkLLmM53q8Ech65eZktRV%2FSzWOQTEOxqA%2B%2B5xNur30TsymFgav3XUb4yLJ%2B6ElpLlhoNTUVc6XAF%2F%2BYzfqw6NIQYoWNq3eQloOg%2FifSyRavOmuQVL8fnT51VQSkXdtFgFsWmKqr%2B2xIH9oLKj0NCi3cFXyXeG10tmHUOzyUnK2PBZ7qG7wTpV9%2Fn13oi4xHf8Xt2Ba5A%2FgM5j3IpV9oKBCoyp4xmJavIW6uEPCpFH2at5mL5OFOqyQPlFfEmYMolfyAFYcYDGdxg%3D%3D HTTP/1.1
Host: 2d0e76e05bb785204f4f720a65f6281b.r2.cloudflarestorage.com
User-Agent: rclone/v1.59.2
Content-Length: 5242880
Authorization: XXXX
Content-Md5: jsU8QvXBI8kact57UykwXw==
Expect: 100-Continue
X-Amz-Content-Sha256: ac041f502d5d4b19b52c129d4d28404d063cf8f5a467eff17a786d6dd639115e
X-Amz-Date: 20220928T214709Z
Accept-Encoding: gzip

2022/09/28 21:47:09 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/09/28 21:47:09 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/09/28 21:47:09 DEBUG : HTTP RESPONSE (req 0xc000d28a00)
2022/09/28 21:47:09 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Cf-Ray: 751fad050a1b57a0-IAD
Connection: keep-alive
Date: Wed, 28 Sep 2022 21:47:09 GMT
Etag: "ACgrl+QFs3Xa4fYPULXR0ax2G7ehqPh+CLcl9wZMqYDOx+fCsIKtgiGlxAR+YdXDHqlzagzNPhQgujkAxuyNAfQuZsDNGadpV2ElSvDJh9UP0wxBg2+I8qajW0Vp3sICjyIR5LTc7XwY4dlfVN0Gu5eNaYPCjqX4hekNGPrifMGQju7wZbdQ91bNnsxlyp2g+q+UlAnmpL0stdX5gIviAXOHuNB2gr1whDU2UDzW4R+ss6xEwTzgtIY4jpF+JAN55w=="
Server: cloudflare
Vary: Accept-Encoding

2022/09/28 21:47:09 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/09/28 21:47:09 DEBUG : tempfile: multipart upload starting chunk 6 size 5Mi offset 25Mi/1Gi
2022/09/28 21:47:09 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/09/28 21:47:09 DEBUG : HTTP REQUEST (req 0xc000d28b00)
2022/09/28 21:47:09 DEBUG : PUT /temp2/tempfile?partNumber=6&uploadId=ABA5HzhZKiobirEUckHtYFA8cgWGX%2FLPGS8IL%2FEringaHZRqWsOD%2BNIkLLmM53q8Ech65eZktRV%2FSzWOQTEOxqA%2B%2B5xNur30TsymFgav3XUb4yLJ%2B6ElpLlhoNTUVc6XAF%2F%2BYzfqw6NIQYoWNq3eQloOg%2FifSyRavOmuQVL8fnT51VQSkXdtFgFsWmKqr%2B2xIH9oLKj0NCi3cFXyXeG10tmHUOzyUnK2PBZ7qG7wTpV9%2Fn13oi4xHf8Xt2Ba5A%2FgM5j3IpV9oKBCoyp4xmJavIW6uEPCpFH2at5mL5OFOqyQPlFfEmYMolfyAFYcYDGdxg%3D%3D HTTP/1.1
Host: 2d0e76e05bb785204f4f720a65f6281b.r2.cloudflarestorage.com
User-Agent: rclone/v1.59.2
Content-Length: 5242880
Authorization: XXXX
Content-Md5: zkXb9aAJ4Uu1JIgINAHIqw==
Expect: 100-Continue
X-Amz-Content-Sha256: b630e87b05091e7b01cfebc39e71d35c853a850c1df469acd6eb0b4a09e54d55
X-Amz-Date: 20220928T214709Z
Accept-Encoding: gzip

2022/09/28 21:47:09 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/09/28 21:47:09 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/09/28 21:47:09 DEBUG : HTTP RESPONSE (req 0xc00027e500)
2022/09/28 21:47:09 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Cf-Ray: 751fad04ff18829f-IAD
Connection: keep-alive
Date: Wed, 28 Sep 2022 21:47:09 GMT
Etag: "AD69h3XnJzhvpCF4CkBVs8mVk0ScEH6pPLTHdc2RtRoY+r1U/tcoeT8vB0ko8IfId6P2YIY+zx1C41Ao/ottKGK3n8UX8nqAUyi+fa3qI6RGlgBnrUfy7XdqhL/JUG7Gvc9nlohEhYFIxXSKWU7v8tOIPu+mS9z55nFHlW1Z6Q9zuRxQQ2avqd6MMR02VQPnwY/e8Is9W25hb5O36jmCicutPUIGGj10abB4FMf0mGvYuW5+++++fTw3IPJx1gJgJQ=="
Server: cloudflare
Vary: Accept-Encoding

2022/09/28 21:47:09 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/09/28 21:47:09 DEBUG : tempfile: multipart upload starting chunk 7 size 5Mi offset 30Mi/1Gi
2022/09/28 21:47:09 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/09/28 21:47:09 DEBUG : HTTP REQUEST (req 0xc000e16300)
2022/09/28 21:47:09 DEBUG : PUT /temp2/tempfile?partNumber=7&uploadId=ABA5HzhZKiobirEUckHtYFA8cgWGX%2FLPGS8IL%2FEringaHZRqWsOD%2BNIkLLmM53q8Ech65eZktRV%2FSzWOQTEOxqA%2B%2B5xNur30TsymFgav3XUb4yLJ%2B6ElpLlhoNTUVc6XAF%2F%2BYzfqw6NIQYoWNq3eQloOg%2FifSyRavOmuQVL8fnT51VQSkXdtFgFsWmKqr%2B2xIH9oLKj0NCi3cFXyXeG10tmHUOzyUnK2PBZ7qG7wTpV9%2Fn13oi4xHf8Xt2Ba5A%2FgM5j3IpV9oKBCoyp4xmJavIW6uEPCpFH2at5mL5OFOqyQPlFfEmYMolfyAFYcYDGdxg%3D%3D HTTP/1.1
Host: 2d0e76e05bb785204f4f720a65f6281b.r2.cloudflarestorage.com
User-Agent: rclone/v1.59.2
Content-Length: 5242880
Authorization: XXXX
Content-Md5: Ctjp5vaNYLs5LV1Q6Hh/kQ==
Expect: 100-Continue
X-Amz-Content-Sha256: 83384222e4016ed7b629b0c5351c4e40fc301b84f421da3ee602023e13043ea9
X-Amz-Date: 20220928T214709Z
Accept-Encoding: gzip

2022/09/28 21:47:09 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/09/28 21:47:09 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/09/28 21:47:09 DEBUG : HTTP RESPONSE (req 0xc0005b9500)
2022/09/28 21:47:09 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Cf-Ray: 751fad078bfd81c4-IAD
Connection: keep-alive
Date: Wed, 28 Sep 2022 21:47:09 GMT
Etag: "AMZgCP6LPfLB7Pwmp1cNBDJVjQANB2Li7q9q80XgcPybEypuP9F7x3liab9h44p3kzgi1P10wTuNapkcZJSO7jTfpNdseZiq6lmFJcoo9KKMG3Yez6uANVp3yzB5qQkZSwd5aCjyftGj4aeVtUCBB27P8ezx/GQwiMmB6sDwHlA4f9ge20a20YWoNaUs8fMWDuxUo5t+KANqyE4uODmLUf6GCp4h0oAXYw5YQqavJVH2s/PzTwFA02tN17kr1FTc0Q=="
Server: cloudflare
Vary: Accept-Encoding

2022/09/28 21:47:09 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/09/28 21:47:09 DEBUG : tempfile: multipart upload starting chunk 8 size 5Mi offset 35Mi/1Gi
2022/09/28 21:47:09 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/09/28 21:47:09 DEBUG : HTTP REQUEST (req 0xc000b84800)
2022/09/28 21:47:09 DEBUG : PUT /temp2/tempfile?partNumber=8&uploadId=ABA5HzhZKiobirEUckHtYFA8cgWGX%2FLPGS8IL%2FEringaHZRqWsOD%2BNIkLLmM53q8Ech65eZktRV%2FSzWOQTEOxqA%2B%2B5xNur30TsymFgav3XUb4yLJ%2B6ElpLlhoNTUVc6XAF%2F%2BYzfqw6NIQYoWNq3eQloOg%2FifSyRavOmuQVL8fnT51VQSkXdtFgFsWmKqr%2B2xIH9oLKj0NCi3cFXyXeG10tmHUOzyUnK2PBZ7qG7wTpV9%2Fn13oi4xHf8Xt2Ba5A%2FgM5j3IpV9oKBCoyp4xmJavIW6uEPCpFH2at5mL5OFOqyQPlFfEmYMolfyAFYcYDGdxg%3D%3D HTTP/1.1
Host: 2d0e76e05bb785204f4f720a65f6281b.r2.cloudflarestorage.com
User-Agent: rclone/v1.59.2
Content-Length: 5242880
Authorization: XXXX
Content-Md5: 7Ynr/ZK0cvDKwW0lOl1D6A==
Expect: 100-Continue
X-Amz-Content-Sha256: dee91d5925c56fe32879a4ce249c2ddb48eee73099c800cddfc8d40a8ba45c9b
X-Amz-Date: 20220928T214709Z
Accept-Encoding: gzip

2022/09/28 21:47:09 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/09/28 21:47:09 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/09/28 21:47:09 DEBUG : HTTP RESPONSE (req 0xc000b84600)
2022/09/28 21:47:09 DEBUG : HTTP/1.1 503 Service Unavailable
Content-Length: 159
Cf-Ray: 751fad050cec5a21-IAD
Connection: keep-alive
Content-Type: application/xml
Date: Wed, 28 Sep 2022 21:47:09 GMT
Server: cloudflare
Vary: Accept-Encoding

<?xml version="1.0" encoding="UTF-8"?><Error><Code>ServiceUnavailable</Code><Message>Reduce your concurrent request rate for the same object.</Message></Error>
2022/09/28 21:47:09 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/09/28 21:47:10 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/09/28 21:47:10 DEBUG : HTTP RESPONSE (req 0xc000d28b00)
2022/09/28 21:47:10 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Cf-Ray: 751fad081e6a57a0-IAD
Connection: keep-alive
Date: Wed, 28 Sep 2022 21:47:09 GMT
Etag: "ALJKBWsAspFXleXK+wQHiJTaN6BKnWgsDnLP6DDX1/6intDjHlBPcwlD0F6GXxx5q4Alv4KELMEt09U91m+T5+eHHUwzqO+SXT/wR6CFlu2Bufhfzg8cKMjvOmv4gSjQ1kXMumHlVoRlAA3H8mYG33qTwMHG4PRr4wU2ukA/WdeEHLdPxlDOFpxG/yvsFzjS71TQQJwqJHadWa2g9AXUuwOLwCpmCg8j9Y+TqK17oIfP8fl68oGwpf/mb80XNmEGqA=="
Server: cloudflare
Vary: Accept-Encoding

2022/09/28 21:47:10 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/09/28 21:47:10 DEBUG : tempfile: multipart upload starting chunk 9 size 5Mi offset 40Mi/1Gi
2022/09/28 21:47:10 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/09/28 21:47:10 DEBUG : HTTP REQUEST (req 0xc000b84e00)
2022/09/28 21:47:10 DEBUG : PUT /temp2/tempfile?partNumber=9&uploadId=ABA5HzhZKiobirEUckHtYFA8cgWGX%2FLPGS8IL%2FEringaHZRqWsOD%2BNIkLLmM53q8Ech65eZktRV%2FSzWOQTEOxqA%2B%2B5xNur30TsymFgav3XUb4yLJ%2B6ElpLlhoNTUVc6XAF%2F%2BYzfqw6NIQYoWNq3eQloOg%2FifSyRavOmuQVL8fnT51VQSkXdtFgFsWmKqr%2B2xIH9oLKj0NCi3cFXyXeG10tmHUOzyUnK2PBZ7qG7wTpV9%2Fn13oi4xHf8Xt2Ba5A%2FgM5j3IpV9oKBCoyp4xmJavIW6uEPCpFH2at5mL5OFOqyQPlFfEmYMolfyAFYcYDGdxg%3D%3D HTTP/1.1
Host: 2d0e76e05bb785204f4f720a65f6281b.r2.cloudflarestorage.com
User-Agent: rclone/v1.59.2
Content-Length: 5242880
Authorization: XXXX
Content-Md5: oiX/GtrajVexVGWto5px5A==
Expect: 100-Continue
X-Amz-Content-Sha256: 6de6d55507226f16847cc3f1ecbac74408b3b59c3584816bb06c7bc60a44ed08
X-Amz-Date: 20220928T214710Z
Accept-Encoding: gzip

2022/09/28 21:47:10 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/09/28 21:47:10 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/09/28 21:47:10 DEBUG : HTTP RESPONSE (req 0xc000e16300)
2022/09/28 21:47:10 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Cf-Ray: 751fad08cf10829f-IAD
Connection: keep-alive
Date: Wed, 28 Sep 2022 21:47:10 GMT
Etag: "AO29V7f/xGe7MpFqwQOZAEXm1fQ9AxXbm6Q63qsSwHHfxpXdtIMGJChC9Rq/D4xhqsfnSKWiO1mMJxHASP1Oll5NI0xG8gjkdMXWISBUKEgzIOYK/OmdH67B18XHnP/LQ5brdDjJVHy1OcYjHvzv/ZlW/OLo4XDcfCL3kmcRF67xWE8QRaSeRznba57TI3FMhqMgTVd24F/q726nQCOEHXWRe58tuOZil5yiEoAkvFzDdDgisjUDdccDP5cnlNkYjA=="
Server: cloudflare
Vary: Accept-Encoding

2022/09/28 21:47:10 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/09/28 21:47:10 DEBUG : tempfile: multipart upload starting chunk 10 size 5Mi offset 45Mi/1Gi
2022/09/28 21:47:10 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/09/28 21:47:10 DEBUG : HTTP REQUEST (req 0xc00085c300)
2022/09/28 21:47:10 DEBUG : PUT /temp2/tempfile?partNumber=10&uploadId=ABA5HzhZKiobirEUckHtYFA8cgWGX%2FLPGS8IL%2FEringaHZRqWsOD%2BNIkLLmM53q8Ech65eZktRV%2FSzWOQTEOxqA%2B%2B5xNur30TsymFgav3XUb4yLJ%2B6ElpLlhoNTUVc6XAF%2F%2BYzfqw6NIQYoWNq3eQloOg%2FifSyRavOmuQVL8fnT51VQSkXdtFgFsWmKqr%2B2xIH9oLKj0NCi3cFXyXeG10tmHUOzyUnK2PBZ7qG7wTpV9%2Fn13oi4xHf8Xt2Ba5A%2FgM5j3IpV9oKBCoyp4xmJavIW6uEPCpFH2at5mL5OFOqyQPlFfEmYMolfyAFYcYDGdxg%3D%3D HTTP/1.1
Host: 2d0e76e05bb785204f4f720a65f6281b.r2.cloudflarestorage.com
User-Agent: rclone/v1.59.2
Content-Length: 5242880
Authorization: XXXX
Content-Md5: jmHyq5K/3ylI8sXZEVrsbQ==
Expect: 100-Continue
X-Amz-Content-Sha256: 7aebecf071226dcd8c74b9cbec5fad762b879b8fdd88e48b024c5514af19c002
X-Amz-Date: 20220928T214710Z
Accept-Encoding: gzip

2022/09/28 21:47:10 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/09/28 21:47:10 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/09/28 21:47:10 DEBUG : HTTP RESPONSE (req 0xc000b84800)
2022/09/28 21:47:10 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Cf-Ray: 751fad0b1a6a81c4-IAD
Connection: keep-alive
Date: Wed, 28 Sep 2022 21:47:10 GMT
Etag: "ADzJnwNuJlYz7bqX5XVcfFwcxfRHtWldm39fYfJSxkwkwdEnqTo6CPhQDhaf3OyoRk1DobSTpd/BBhcP+7fFbOLKuWi1GOG/hYDLOTWu+IqHyy0C3VJ2oPnpswx8G2LZedXfxaXSAFNR7ue2fbh4X1EJTJQEiGj5k0viMv7LzvCkVPuRMZyt62e1CIAZOXoi8P5RUHopchR7PKjyzi4vyF/MwQOk3HiKiUu1bAvUTc53eFHRBnjLxgfXOzqNp4Sx4g=="
Server: cloudflare
Vary: Accept-Encoding

2022/09/28 21:47:10 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/09/28 21:47:10 DEBUG : tempfile: multipart upload starting chunk 11 size 5Mi offset 50Mi/1Gi
2022/09/28 21:47:10 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/09/28 21:47:10 DEBUG : HTTP REQUEST (req 0xc00085c200)
2022/09/28 21:47:10 DEBUG : PUT /temp2/tempfile?partNumber=11&uploadId=ABA5HzhZKiobirEUckHtYFA8cgWGX%2FLPGS8IL%2FEringaHZRqWsOD%2BNIkLLmM53q8Ech65eZktRV%2FSzWOQTEOxqA%2B%2B5xNur30TsymFgav3XUb4yLJ%2B6ElpLlhoNTUVc6XAF%2F%2BYzfqw6NIQYoWNq3eQloOg%2FifSyRavOmuQVL8fnT51VQSkXdtFgFsWmKqr%2B2xIH9oLKj0NCi3cFXyXeG10tmHUOzyUnK2PBZ7qG7wTpV9%2Fn13oi4xHf8Xt2Ba5A%2FgM5j3IpV9oKBCoyp4xmJavIW6uEPCpFH2at5mL5OFOqyQPlFfEmYMolfyAFYcYDGdxg%3D%3D HTTP/1.1
Host: 2d0e76e05bb785204f4f720a65f6281b.r2.cloudflarestorage.com
User-Agent: rclone/v1.59.2
Content-Length: 5242880
Authorization: XXXX
Content-Md5: HLo4t1YvyyZK5illLAKZ5Q==
Expect: 100-Continue
X-Amz-Content-Sha256: c14a403fd24704612fdd72f02ea63d2128f8a8df9782aa5bab6146a97f36cab1
X-Amz-Date: 20220928T214710Z
Accept-Encoding: gzip

2022/09/28 21:47:10 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/09/28 21:47:10 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/09/28 21:47:10 DEBUG : HTTP RESPONSE (req 0xc000b84e00)
2022/09/28 21:47:10 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Cf-Ray: 751fad0babea57a0-IAD
Connection: keep-alive
Date: Wed, 28 Sep 2022 21:47:10 GMT
Etag: "AHiD8pPnW0Qob7uFxwn3HjrGAF3PNfQMDPfe633/WylV8u6/rlZnZ0T3jIvqFtbdtseaHO040z8zDysRzISQDdP5ScxmRYJtO4VTN/0hQKVC8lUVDN3mIKuDDJToHUyE7x2tFrT0u8jWATAs9lfQLShIkLwEdkZUxugd1aWlBrpRB0uf/lo3++vFVkN1QNL4AsWODVnKE0u4POM8f4iFvau5yEUIfHIB7QdxBarhI+cZBI0k5YQc43m3i4Tmo6ig6g=="
Server: cloudflare
Vary: Accept-Encoding

2022/09/28 21:47:10 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/09/28 21:47:10 DEBUG : tempfile: multipart upload starting chunk 12 size 5Mi offset 55Mi/1Gi
2022/09/28 21:47:10 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/09/28 21:47:10 DEBUG : HTTP REQUEST (req 0xc000b84a00)
2022/09/28 21:47:10 DEBUG : PUT /temp2/tempfile?partNumber=12&uploadId=ABA5HzhZKiobirEUckHtYFA8cgWGX%2FLPGS8IL%2FEringaHZRqWsOD%2BNIkLLmM53q8Ech65eZktRV%2FSzWOQTEOxqA%2B%2B5xNur30TsymFgav3XUb4yLJ%2B6ElpLlhoNTUVc6XAF%2F%2BYzfqw6NIQYoWNq3eQloOg%2FifSyRavOmuQVL8fnT51VQSkXdtFgFsWmKqr%2B2xIH9oLKj0NCi3cFXyXeG10tmHUOzyUnK2PBZ7qG7wTpV9%2Fn13oi4xHf8Xt2Ba5A%2FgM5j3IpV9oKBCoyp4xmJavIW6uEPCpFH2at5mL5OFOqyQPlFfEmYMolfyAFYcYDGdxg%3D%3D HTTP/1.1
Host: 2d0e76e05bb785204f4f720a65f6281b.r2.cloudflarestorage.com
User-Agent: rclone/v1.59.2
Content-Length: 5242880
Authorization: XXXX
Content-Md5: kIOew0Lz7w2WdnGBvTFO5Q==
Expect: 100-Continue
X-Amz-Content-Sha256: 6c6abf65c477a552e8f27a84ced5146b2a77968a3b160bd0d83d44a28fba517a
X-Amz-Date: 20220928T214710Z
Accept-Encoding: gzip

2022/09/28 21:47:10 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/09/28 21:47:10 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/09/28 21:47:10 DEBUG : HTTP REQUEST (req 0xc00085c100)
2022/09/28 21:47:10 DEBUG : PUT /temp2/tempfile?partNumber=2&uploadId=ABA5HzhZKiobirEUckHtYFA8cgWGX%2FLPGS8IL%2FEringaHZRqWsOD%2BNIkLLmM53q8Ech65eZktRV%2FSzWOQTEOxqA%2B%2B5xNur30TsymFgav3XUb4yLJ%2B6ElpLlhoNTUVc6XAF%2F%2BYzfqw6NIQYoWNq3eQloOg%2FifSyRavOmuQVL8fnT51VQSkXdtFgFsWmKqr%2B2xIH9oLKj0NCi3cFXyXeG10tmHUOzyUnK2PBZ7qG7wTpV9%2Fn13oi4xHf8Xt2Ba5A%2FgM5j3IpV9oKBCoyp4xmJavIW6uEPCpFH2at5mL5OFOqyQPlFfEmYMolfyAFYcYDGdxg%3D%3D HTTP/1.1
Host: 2d0e76e05bb785204f4f720a65f6281b.r2.cloudflarestorage.com
User-Agent: rclone/v1.59.2
Content-Length: 5242880
Authorization: XXXX
Content-Md5: rVE7WDKzXzcPz3fE4Cke4w==
Expect: 100-Continue
X-Amz-Content-Sha256: 449fa97d0b26b3457e6a845c22fc4f7b70fb338d4320c8f951c5ca34e898ab89
X-Amz-Date: 20220928T214710Z
Accept-Encoding: gzip

2022/09/28 21:47:10 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/09/28 21:47:10 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/09/28 21:47:10 DEBUG : HTTP RESPONSE (req 0xc00085c100)
2022/09/28 21:47:10 DEBUG : HTTP/1.1 403 Forbidden
Transfer-Encoding: chunked
Cf-Ray: 751fad0f8c2c5a21-IAD
Connection: keep-alive
Content-Type: application/xml
Date: Wed, 28 Sep 2022 21:47:10 GMT
Server: cloudflare
Vary: Accept-Encoding

ea
<?xml version="1.0" encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your secret access key and signing method. </Message></Error>
0

2022/09/28 21:47:10 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/09/28 21:47:10 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/09/28 21:47:10 DEBUG : HTTP RESPONSE (req 0xc00085c300)
2022/09/28 21:47:10 DEBUG : Error: context canceled
2022/09/28 21:47:10 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/09/28 21:47:10 DEBUG : pacer: low level retry 1/2 (error RequestCanceled: request context canceled
caused by: context canceled)
2022/09/28 21:47:10 DEBUG : pacer: Rate limited, increasing sleep to 10ms
2022/09/28 21:47:10 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/09/28 21:47:10 DEBUG : HTTP RESPONSE (req 0xc000b84a00)
2022/09/28 21:47:10 DEBUG : Error: context canceled
2022/09/28 21:47:10 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/09/28 21:47:10 DEBUG : pacer: low level retry 1/2 (error RequestCanceled: request context canceled
caused by: context canceled)
2022/09/28 21:47:10 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2022/09/28 21:47:10 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/09/28 21:47:10 DEBUG : HTTP RESPONSE (req 0xc00085c200)
2022/09/28 21:47:10 DEBUG : Error: context canceled
2022/09/28 21:47:10 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/09/28 21:47:10 DEBUG : pacer: low level retry 1/2 (error RequestCanceled: request context canceled
caused by: context canceled)
2022/09/28 21:47:10 DEBUG : pacer: Rate limited, increasing sleep to 40ms
2022/09/28 21:47:10 DEBUG : pacer: low level retry 2/2 (error RequestCanceled: request context canceled
caused by: context canceled)
2022/09/28 21:47:10 DEBUG : pacer: Rate limited, increasing sleep to 80ms
2022/09/28 21:47:10 DEBUG : pacer: low level retry 2/2 (error RequestCanceled: request context canceled
caused by: context canceled)
2022/09/28 21:47:10 DEBUG : pacer: Rate limited, increasing sleep to 160ms
2022/09/28 21:47:10 DEBUG : pacer: low level retry 2/2 (error RequestCanceled: request context canceled
caused by: context canceled)
2022/09/28 21:47:10 DEBUG : pacer: Rate limited, increasing sleep to 320ms
2022/09/28 21:47:10 DEBUG : tempfile: Cancelling multipart upload
2022/09/28 21:47:10 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/09/28 21:47:10 DEBUG : HTTP REQUEST (req 0xc000b85100)
2022/09/28 21:47:10 DEBUG : DELETE /temp2/tempfile?uploadId=ABA5HzhZKiobirEUckHtYFA8cgWGX%2FLPGS8IL%2FEringaHZRqWsOD%2BNIkLLmM53q8Ech65eZktRV%2FSzWOQTEOxqA%2B%2B5xNur30TsymFgav3XUb4yLJ%2B6ElpLlhoNTUVc6XAF%2F%2BYzfqw6NIQYoWNq3eQloOg%2FifSyRavOmuQVL8fnT51VQSkXdtFgFsWmKqr%2B2xIH9oLKj0NCi3cFXyXeG10tmHUOzyUnK2PBZ7qG7wTpV9%2Fn13oi4xHf8Xt2Ba5A%2FgM5j3IpV9oKBCoyp4xmJavIW6uEPCpFH2at5mL5OFOqyQPlFfEmYMolfyAFYcYDGdxg%3D%3D HTTP/1.1
Host: 2d0e76e05bb785204f4f720a65f6281b.r2.cloudflarestorage.com
User-Agent: rclone/v1.59.2
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220928T214710Z
Accept-Encoding: gzip

2022/09/28 21:47:10 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/09/28 21:47:11 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/09/28 21:47:11 DEBUG : HTTP RESPONSE (req 0xc000b85100)
2022/09/28 21:47:11 DEBUG : HTTP/1.1 204 No Content
Cf-Ray: 751fad11595f0674-IAD
Connection: keep-alive
Date: Wed, 28 Sep 2022 21:47:11 GMT
Server: cloudflare
Vary: Accept-Encoding

2022/09/28 21:47:11 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/09/28 21:47:11 DEBUG : pacer: Reducing sleep to 240ms
2022/09/28 21:47:11 ERROR : tempfile: Failed to copy: multipart upload failed to upload part: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method. 
	status code: 403, request id: , host id: 
2022/09/28 21:47:11 ERROR : Attempt 1/1 failed with 1 errors and: multipart upload failed to upload part: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method. 
	status code: 403, request id: , host id: 
2022/09/28 21:47:11 INFO  : 
Transferred:   	       60 MiB / 60 MiB, 100%, 13.747 MiB/s, ETA 0s
Errors:                 1 (retrying may help)
Elapsed time:         5.1s

2022/09/28 21:47:11 DEBUG : 9 go routines active
2022/09/28 21:47:11 Failed to copy: multipart upload failed to upload part: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method. 
	status code: 403, request id: , host id:
6

Managed to upload the file by using:

--s3-upload-concurrency 2

However, that does reduce the upload speed.

7

Perhaps you can compensate by doubling --transfers, try adding --transfers=8 to your command.

More info here:
https://rclone.org/s3/#multipart-uploads
https://rclone.org/docs/#transfers-n

8

That log has this in it when uploading part 8

2022/09/28 21:47:09 DEBUG : HTTP RESPONSE (req 0xc000b84600)
2022/09/28 21:47:09 DEBUG : HTTP/1.1 503 Service Unavailable
Content-Length: 159
Cf-Ray: 751fad050cec5a21-IAD
Connection: keep-alive
Content-Type: application/xml
Date: Wed, 28 Sep 2022 21:47:09 GMT
Server: cloudflare
Vary: Accept-Encoding

<?xml version="1.0" encoding="UTF-8"?><Error><Code>ServiceUnavailable</Code><Message>Reduce your concurrent request rate for the same object.</Message></Error>

So it looks like reducing the concurrency is the correct thing to do.

I'm not sure why cloudflare then gave the signature mis-match error - maybe it is a way of getting rclone to really stop, because it will retry 503 errors normally.

9

Linking this to the relevant GH issue for my / future people's convenience: [S3] "Failed to copy: multipart upload failed to upload part" using Cloudflare provider · Issue #6193 · rclone/rclone · GitHub

10

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.