RClone and Cloudflare R2 issues

Suspected Bug
1

What is the problem you are having with rclone?

RClone is unable to complete large copies to Cloudflare R2.

Copying files greater than 5GB seems to trigger the below error consistently.

SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.
status code: 403, request id: , host id:

It will retry 3 times and abort eventually at different points.

I've tried different chunk sizes, transfers, s3-upload-cutoff sizes to no avail. Not sure if it's a cloudflare issue.

Run the command 'rclone version' and share the full output of the command.

rclone v1.59.0-beta.6142.542c1616b

  • os/version: debian 10.12 (64 bit)
  • os/kernel: 5.4.174-2-pve (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.18.2
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Cloudflare R2

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy vzdump-qemu-103-2022_05_13-20_51_40.vma.zst r2:prox1-bk/test/1 -vv --stats=5s```


#### The rclone config contents with secrets removed.  
<!--  You should use 3 backticks to begin and end your paste to make it readable.   -->

[r2]
type = s3
provider = Cloudflare
access_key_id = access
secret_access_key = mysecret
region = auto
endpoint = https://myaccount.r2.cloudflarestorage.com
acl = private




#### A log from the command with the `-vv` flag  
<!-- You should use 3 backticks to begin and end your paste to make it readable.  Or use a service such as https://pastebin.com or https://gist.github.com/   -->

root@cl-tx-1:/media/vz/dump# rclone copy vzdump-qemu-103-2022_05_13-20_51_40.vma.zst r2:prox1-bk/test/1 -vv --stats=5s
2022/05/13 21:20:07 DEBUG : rclone: Version "v1.59.0-beta.6142.542c1616b" starting with parameters ["rclone" "copy" "vzdump-qemu-103-2022_05_13-20_51_40.vma.zst" "r2:prox1-bk/test/1" "-vv" "--stats=5s"]
2022/05/13 21:20:07 DEBUG : Creating backend with remote "vzdump-qemu-103-2022_05_13-20_51_40.vma.zst"
2022/05/13 21:20:07 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2022/05/13 21:20:07 DEBUG : fs cache: adding new entry for parent of "vzdump-qemu-103-2022_05_13-20_51_40.vma.zst", "/media/vz/dump"
2022/05/13 21:20:07 DEBUG : Creating backend with remote "r2:prox1-bk/test/1"
2022/05/13 21:20:09 DEBUG : vzdump-qemu-103-2022_05_13-20_51_40.vma.zst: Need to transfer - File not found at Destination
2022/05/13 21:20:13 INFO :
2022/05/13 21:21:32 DEBUG : vzdump-qemu-103-2022_05_13-20_51_40.vma.zst: multipart upload starting chunk 4 size 5Mi offset 15Mi/14.513Gi
2022/05/13 21:21:33 INFO :
Transferred: 20 MiB / 14.513 GiB, 0%, 1.251 MiB/s, ETA 3h17m46s
Transferred: 0 / 1, 0%
Elapsed time: 1m26.0s
Transferring:

  • vzdump-qemu-103-2022_05_13-20_51_40.vma.zst: 0% /14.513Gi, 1.251Mi/s, 3h17m46s

2022/05/13 21:21:34 DEBUG : vzdump-qemu-103-2022_05_13-20_51_40.vma.zst: multipart upload starting chunk 5 size 5Mi offset 20Mi/14.513Gi
2022/05/13 21:21:34 DEBUG : vzdump-qemu-103-2022_05_13-20_51_40.vma.zst: multipart upload starting chunk 6 size 5Mi offset 25Mi/14.513Gi
2022/05/13 21:21:35 DEBUG : vzdump-qemu-103-2022_05_13-20_51_40.vma.zst: multipart upload starting chunk 7 size 5Mi offset 30Mi/14.513Gi
2022/05/13 21:21:35 DEBUG : vzdump-qemu-103-2022_05_13-20_51_40.vma.zst: multipart upload starting chunk 8 size 5Mi offset 35Mi/14.513Gi
2022/05/13 21:21:35 DEBUG : vzdump-qemu-103-2022_05_13-20_51_40.vma.zst: multipart upload starting chunk 9 size 5Mi offset 40Mi/14.513Gi
2022/05/13 21:21:35 DEBUG : vzdump-qemu-103-2022_05_13-20_51_40.vma.zst: multipart upload starting chunk 10 size 5Mi offset 45Mi/14.513Gi
2022/05/13 21:21:36 DEBUG : vzdump-qemu-103-2022_05_13-20_51_40.vma.zst: multipart upload starting chunk 11 size 5Mi offset 50Mi/14.513Gi
2022/05/13 21:21:36 DEBUG : vzdump-qemu-103-2022_05_13-20_51_40.vma.zst: multipart upload starting chunk 12 size 5Mi offset 55Mi/14.513Gi
2022/05/13 21:21:36 DEBUG : vzdump-qemu-103-2022_05_13-20_51_40.vma.zst: multipart upload starting chunk 13 size 5Mi offset 60Mi/14.513Gi
2022/05/13 21:21:36 DEBUG : vzdump-qemu-103-2022_05_13-20_51_40.vma.zst: multipart upload starting chunk 14 size 5Mi offset 65Mi/14.513Gi
2022/05/13 21:21:37 DEBUG : pacer: low level retry 1/2 (error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method. **
** status code: 403, request id: , host id: )
2022/05/13 21:21:37 DEBUG : pacer: Rate limited, increasing sleep to 10ms
2022/05/13 21:21:37 DEBUG : pacer: Reducing sleep to 0s
2022/05/13 21:21:37 DEBUG : vzdump-qemu-103-2022_05_13-20_51_40.vma.zst: multipart upload starting chunk 15 size 5Mi offset 70Mi/14.513Gi
2022/05/13 21:21:37 DEBUG : vzdump-qemu-103-2022_05_13-20_51_40.vma.zst: multipart upload starting chunk 16 size 5Mi offset 75Mi/14.513Gi
2022/05/13 21:21:38 DEBUG : vzdump-qemu-103-2022_05_13-20_51_40.vma.zst: multipart upload starting chunk 17 size 5Mi offset 80Mi/14.513Gi
2022/05/13 21:21:38 DEBUG : vzdump-qemu-103-2022_05_13-20_51_40.vma.zst: multipart upload starting chunk 18 size 5Mi offset 85Mi/14.513Gi
2022/05/13 21:21:38 DEBUG : vzdump-qemu-103-2022_05_13-20_51_40.vma.zst: multipart upload starting chunk 19 size 5Mi offset 90Mi/14.513Gi
2022/05/13 21:21:38 INFO :
Transferred: 95 MiB / 14.513 GiB, 1%, 4.245 MiB/s, ETA 57m58s
Transferred: 0 / 1, 0%
Elapsed time: 1m31.0s
Transferring:

  • vzdump-qemu-103-2022_05_13-20_51_40.vma.zst: 0% /14.513Gi, 4.245Mi/s, 57m58s

2022/05/13 21:21:39 DEBUG : vzdump-qemu-103-2022_05_13-20_51_40.vma.zst: multipart upload starting chunk 20 size 5Mi offset 95Mi/14.513Gi
2022/05/13 21:21:39 DEBUG : vzdump-qemu-103-2022_05_13-20_51_40.vma.zst: multipart upload starting chunk 21 size 5Mi offset 100Mi/14.513Gi
2022/05/13 21:21:39 DEBUG : vzdump-qemu-103-2022_05_13-20_51_40.vma.zst: multipart upload starting chunk 22 size 5Mi offset 105Mi/14.513Gi
2022/05/13 21:21:40 DEBUG : vzdump-qemu-103-2022_05_13-20_51_40.vma.zst: multipart upload starting chunk 23 size 5Mi offset 110Mi/14.513Gi
.....
....

2

hello and welcome to the forum,

In testing, I think that this is a cloudflare issue.
keep in mind that
--- R2 is beta,
--- R2 documentation states that, during the beta, the maximum number transactions per second are rate limited.
--- each provider has it own quirks, requires its own set of tweaks.

i have found that this combination of tweaked flags works around the errors.
you might need to re-tweak the settings based on your internet connection speeds.

--tps-limit=5
--s3-chunk-size=256M

and might need to tweak --s3-upload-concurrency

1 Like
3

I think the the ultimate cause of that error message is rate limiting as stated above.

The actual error message returned is a little odd though, I don't know why retrying a chunk would get signature mismatch.

That probably is a bug in rclone. Fixing it won't solve this problem though!

1 Like
4

Thank you very much. --tpslimit=5 seems to have done it. Running a few more tests but managed to up a 15GB file which I wasn't able to before.

5

good,
i think the goal is to make cloudflare R2, a first class provider, same as aws and wasabi.

so keep us posted, learn and return!

6

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.