Still getting the SignatureDoesNotMatch problem with Cloudflare r2

nucleon · September 17, 2023, 3:36am

So there are lots of posts on here about problems with using rclone with Cloudflare r2 for very large files. However they all seem to treat it as a "solved" problem. I have absolutely not been able to resolve it despite days of trying.

I have been able to upload trivially small files to r2 with rclone, no problem. Which makes me think that it is NOT some kind of authentication error, but an issue with multipart file uploads.

What happens is that no matter what settings I try, it gives me the dreaded SignatureDoesNotMatch error on chunks of the file.

The file is about 4GB in size. I upped the retries to 1,000 and let it run all night, and it managed to "send" 80GB of data but only 105MB was actually delivered to the file. I have checked every post on here on this. I have set things up the way that Cloudflare recommends. No dice.

Here's my rclone version:

rclone version
rclone v1.64.0
- os/version: Microsoft Windows 10 Pro 20H2 (64 bit)
- os/kernel: 10.0.19042.1055 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.21.1
- go/linking: static
- go/tags: cmount

Here's my rclone.conf:

[r2]
type = s3
provider = Cloudflare
access_key_id = <redacted>
secret_access_key = <redacted>
region = auto
endpoint = https://<redacted>.r2.cloudflarestorage.com
acl = private

Here's an example of a recent attempt I made and the log file output. Following the advice I've seen again and again, I've tried to limit how much data is sent and the rate it is sent.

D:\maptiles>rclone copy nasa.pmtiles r2:osmtiles/ --transfers=1 --tpslimit=1 --s3-upload-concurrency=1 --s3-upload-cutoff=100M --s3-chunk-size=100M -v
2023/09/16 23:13:52 INFO  : Starting transaction limiter: max 1 transactions/s with burst 1
2023/09/16 23:14:22 ERROR : nasa.pmtiles: Failed to copy: multi-thread copy: failed to write chunk: failed to upload chunk 1 with 104857600 bytes: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.
        status code: 403, request id: , host id:
2023/09/16 23:14:22 ERROR : Attempt 1/3 failed with 1 errors and: multi-thread copy: failed to write chunk: failed to upload chunk 1 with 104857600 bytes: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.
        status code: 403, request id: , host id:
2023/09/16 23:14:33 ERROR : nasa.pmtiles: Failed to copy: multi-thread copy: failed to write chunk: failed to upload chunk 1 with 104857600 bytes: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.
        status code: 403, request id: , host id:
2023/09/16 23:14:33 ERROR : Attempt 2/3 failed with 1 errors and: multi-thread copy: failed to write chunk: failed to upload chunk 1 with 104857600 bytes: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.
        status code: 403, request id: , host id:
2023/09/16 23:14:52 INFO  :
Transferred:       95.781 MiB / 3.841 GiB, 2%, 1.834 MiB/s, ETA 34m52s
Transferred:            0 / 1, 0%
Elapsed time:       1m0.0s
Transferring:
 *                                  nasa.pmtiles:  0% /3.781Gi, 2.689Mi/s, 23m47s

2023/09/16 23:15:15 ERROR : nasa.pmtiles: Failed to copy: multi-thread copy: failed to write chunk: failed to upload chunk 1 with 104857600 bytes: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.
        status code: 403, request id: , host id:
2023/09/16 23:15:15 ERROR : Attempt 3/3 failed with 1 errors and: multi-thread copy: failed to write chunk: failed to upload chunk 1 with 104857600 bytes: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.
        status code: 403, request id: , host id:
2023/09/16 23:15:15 INFO  :
Transferred:      153.125 MiB / 153.125 MiB, 100%, 2.247 MiB/s, ETA 0s
Errors:                 1 (retrying may help)
Elapsed time:      1m23.5s

2023/09/16 23:15:15 Failed to copy: multi-thread copy: failed to write chunk: failed to upload chunk 1 with 104857600 bytes: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.
        status code: 403, request id: , host id:

I've tried about every combination of settings I can imagine. No changing in the concurrency, tpslimit, transfers, etc. seems to matter. Always the same errors. (Retrying does not help.) I've tried it with huge chunks, tiny chunks, in the middle chunks. The thing is, this file is only the "smaller" of the two files I want to upload -- the other is +100GB.

Here's what I get with huge chunks (a vain attempt to keep it from spamming Cloudflare):

D:\maptiles>rclone copy nasa.pmtiles r2:osmtiles/ --s3-disable-checksum --s3-chunk-size=1G --s3-upload-cutoff=5G --s3-upload-concurrency=1 --progress -vv
2023/09/16 23:25:16 DEBUG : rclone: Version "v1.64.0" starting with parameters ["rclone" "copy" "nasa.pmtiles" "r2:osmtiles/" "--s3-disable-checksum" "--s3-chunk-size=1G" "--s3-upload-cutoff=5G" "--s3-upload-concurrency=1" "--progress" "-vv"]
2023/09/16 23:25:16 DEBUG : Creating backend with remote "nasa.pmtiles"
2023/09/16 23:25:16 DEBUG : Using config file from "C:\\Users\\User\\AppData\\Roaming\\rclone\\rclone.conf"
2023/09/16 23:25:16 DEBUG : fs cache: adding new entry for parent of "nasa.pmtiles", "//?/D:/maptiles"
2023/09/16 23:25:16 DEBUG : Creating backend with remote "r2:osmtiles/"
2023/09/16 23:25:16 DEBUG : r2: detected overridden config - adding "{F7-Ct}" suffix to name
2023/09/16 23:25:16 DEBUG : Resolving service "s3" region "auto"
2023/09/16 23:25:16 DEBUG : fs cache: renaming cache item "r2:osmtiles/" to be canonical "r2{F7-Ct}:osmtiles"
2023-09-16 23:25:16 DEBUG : nasa.pmtiles: Sizes differ (src 4060292083 vs dst 52428800)
2023-09-16 23:25:32 DEBUG : nasa.pmtiles: open chunk writer: started multipart upload: AFq0lLMFtNCxyeJzhNbWYlVBqsoFAQ1RiLXFj-z_maWc4x_UtCfwLVHgbY9vvetMxenOVv4VkymFIs7p3ypGhU58TpqGnp28A7wDfCUw0_mdGRiPPqADV1-7mSNlyG2e_YZy2ICd2QEDoejjVXnaBvHyPW2EKlYTDX85AbZ-64uCTpz19r7hZD8kJG6rnFiQxP6DIMMkyeYL2WlqOy9vJMvgDREvOI2ifvath0m-wpT0equJjJFtkQUIcrsNVMgrshJZ-wU7OgC5l1eMZJxLOXeWr0RyXGutHabmt0yc6CP1Dm6SMRpr9MXrYAxcXxcB0tF3IzHnSelLXw5VaHeNQZI
2023-09-16 23:25:32 DEBUG : nasa.pmtiles: multi-thread copy: using backend concurrency of 1 instead of --multi-thread-streams 4
2023-09-16 23:25:32 DEBUG : nasa.pmtiles: Starting multi-thread copy with 4 chunks of size 1Gi with 1 parallel streams
2023-09-16 23:25:32 DEBUG : nasa.pmtiles: multi-thread copy: chunk 1/4 (0-1073741824) size 1Gi starting
2023-09-16 23:25:37 DEBUG : nasa.pmtiles: multi-thread copy: chunk 1/4 failed: multi-thread copy: failed to write chunk: failed to upload chunk 1 with 1073741824 bytes: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.
        status code: 403, request id: , host id:
2023-09-16 23:25:37 DEBUG : nasa.pmtiles: multi-thread copy: chunk 2/4 (1073741824-2147483648) size 1Gi starting
2023-09-16 23:25:40 DEBUG : nasa.pmtiles: multi-thread copy: chunk 2/4 failed: multi-thread copy: failed to write chunk: failed to upload chunk 2 with 1073741824 bytes: RequestCanceled: request context canceled
caused by: context canceled
2023-09-16 23:25:40 DEBUG : nasa.pmtiles: multi-thread copy: cancelling transfer on exit
2023-09-16 23:25:40 DEBUG : nasa.pmtiles: multipart upload "AFq0lLMFtNCxyeJzhNbWYlVBqsoFAQ1RiLXFj-z_maWc4x_UtCfwLVHgbY9vvetMxenOVv4VkymFIs7p3ypGhU58TpqGnp28A7wDfCUw0_mdGRiPPqADV1-7mSNlyG2e_YZy2ICd2QEDoejjVXnaBvHyPW2EKlYTDX85AbZ-64uCTpz19r7hZD8kJG6rnFiQxP6DIMMkyeYL2WlqOy9vJMvgDREvOI2ifvath0m-wpT0equJjJFtkQUIcrsNVMgrshJZ-wU7OgC5l1eMZJxLOXeWr0RyXGutHabmt0yc6CP1Dm6SMRpr9MXrYAxcXxcB0tF3IzHnSelLXw5VaHeNQZI" aborted
2023-09-16 23:25:40 ERROR : nasa.pmtiles: Failed to copy: multi-thread copy: failed to write chunk: failed to upload chunk 1 with 1073741824 bytes: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.
        status code: 403, request id: , host id:
2023-09-16 23:25:40 ERROR : Attempt 1/3 failed with 1 errors and: multi-thread copy: failed to write chunk: failed to upload chunk 1 with 1073741824 bytes: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.
        status code: 403, request id: , host id:
2023-09-16 23:25:41 DEBUG : nasa.pmtiles: Sizes differ (src 4060292083 vs dst 52428800)
2023-09-16 23:25:46 DEBUG : nasa.pmtiles: open chunk writer: started multipart upload: ALIoUnDVNTuyolutqGGiqSvPm0VwBKVdAY6yaQ3QBCTgBTRgQsAjz15AgOxVFIBJc-k6eHZdy4UBlHo7UZFBd9FWs6I-eUYQDglw4x11d8BL443AfwqqULmiQGbxyuBT8Wr2uEZcueisP2WEYR3p5vsKP7bvwicAWRltP8jefEKqZ7gzSl95u297vcRfwKI3ViKa1BkWov8MEbRQH6ZMJzvsYbQyYeTWVe2JQE7gSNvjzlVTyGrb9DwtNBu3Do3jnQBvHA5UJu7VweygyKls0UzCPY7Ef6yFvVMj-namp7sgjy9uOTBfPl85RBJGEB1ghTkhGL8XlIVtmjyoO3om6sI
2023-09-16 23:25:46 DEBUG : nasa.pmtiles: multi-thread copy: using backend concurrency of 1 instead of --multi-thread-streams 4
2023-09-16 23:25:46 DEBUG : nasa.pmtiles: Starting multi-thread copy with 4 chunks of size 1Gi with 1 parallel streams
2023-09-16 23:25:46 DEBUG : nasa.pmtiles: multi-thread copy: chunk 1/4 (0-1073741824) size 1Gi starting
2023-09-16 23:26:19 DEBUG : nasa.pmtiles: multi-thread copy: chunk 1/4 failed: multi-thread copy: failed to write chunk: failed to upload chunk 1 with 1073741824 bytes: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.
        status code: 403, request id: , host id:
2023-09-16 23:26:19 DEBUG : nasa.pmtiles: multi-thread copy: chunk 2/4 (1073741824-2147483648) size 1Gi starting
2023-09-16 23:26:23 DEBUG : nasa.pmtiles: multi-thread copy: chunk 2/4 failed: multi-thread copy: failed to write chunk: failed to upload chunk 2 with 1073741824 bytes: RequestCanceled: request context canceled
caused by: context canceled
2023-09-16 23:26:23 DEBUG : nasa.pmtiles: multi-thread copy: cancelling transfer on exit
2023-09-16 23:26:23 DEBUG : nasa.pmtiles: multipart upload "ALIoUnDVNTuyolutqGGiqSvPm0VwBKVdAY6yaQ3QBCTgBTRgQsAjz15AgOxVFIBJc-k6eHZdy4UBlHo7UZFBd9FWs6I-eUYQDglw4x11d8BL443AfwqqULmiQGbxyuBT8Wr2uEZcueisP2WEYR3p5vsKP7bvwicAWRltP8jefEKqZ7gzSl95u297vcRfwKI3ViKa1BkWov8MEbRQH6ZMJzvsYbQyYeTWVe2JQE7gSNvjzlVTyGrb9DwtNBu3Do3jnQBvHA5UJu7VweygyKls0UzCPY7Ef6yFvVMj-namp7sgjy9uOTBfPl85RBJGEB1ghTkhGL8XlIVtmjyoO3om6sI" aborted
2023-09-16 23:26:23 ERROR : nasa.pmtiles: Failed to copy: multi-thread copy: failed to write chunk: failed to upload chunk 1 with 1073741824 bytes: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.
        status code: 403, request id: , host id:
2023-09-16 23:26:23 ERROR : Attempt 2/3 failed with 1 errors and: multi-thread copy: failed to write chunk: failed to upload chunk 1 with 1073741824 bytes: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.
        status code: 403, request id: , host id:
2023-09-16 23:26:23 DEBUG : nasa.pmtiles: Sizes differ (src 4060292083 vs dst 52428800)
2023-09-16 23:26:29 DEBUG : nasa.pmtiles: open chunk writer: started multipart upload: AJYdKGi5ZfdVpmtktRqXke34KURxUy8Rn7iacnfPQ4s5Z_MdpNeM8jzD9S4FheM0YLGrBYROqBQju5iyzkxSdSjvuH1xSBBK2FiQUsTtUSIAgN2-C2igcM9kJbhhSKtpq-9VXuj-qRK6S9MlLkcl2LCxBh9kfraeo4UnelEEo9GyrXjNS7UvM38vvkAZg_PHiTIRqWW2CJgNwJctQ9-0WbTX273P_br4iBllCYfSoFonDI5J6xawdztL-TYZoRGMAy_DKhkudeWFk7C43eFWW078HJ2iwUPF0-6x2tDU8_MumHEkkM5B8A5fyECvNwZGuGNcvTIODhTtsrify-j57s8
2023-09-16 23:26:29 DEBUG : nasa.pmtiles: multi-thread copy: using backend concurrency of 1 instead of --multi-thread-streams 4
2023-09-16 23:26:29 DEBUG : nasa.pmtiles: Starting multi-thread copy with 4 chunks of size 1Gi with 1 parallel streams
2023-09-16 23:26:29 DEBUG : nasa.pmtiles: multi-thread copy: chunk 1/4 (0-1073741824) size 1Gi starting
2023-09-16 23:26:56 DEBUG : pacer: low level retry 1/2 (error RequestError: send request failed
caused by: Put "https://084252fb5c3aefb95e32902d03530612.r2.cloudflarestorage.com/osmtiles/nasa.pmtiles?partNumber=1&uploadId=AJYdKGi5ZfdVpmtktRqXke34KURxUy8Rn7iacnfPQ4s5Z_MdpNeM8jzD9S4FheM0YLGrBYROqBQju5iyzkxSdSjvuH1xSBBK2FiQUsTtUSIAgN2-C2igcM9kJbhhSKtpq-9VXuj-qRK6S9MlLkcl2LCxBh9kfraeo4UnelEEo9GyrXjNS7UvM38vvkAZg_PHiTIRqWW2CJgNwJctQ9-0WbTX273P_br4iBllCYfSoFonDI5J6xawdztL-TYZoRGMAy_DKhkudeWFk7C43eFWW078HJ2iwUPF0-6x2tDU8_MumHEkkM5B8A5fyECvNwZGuGNcvTIODhTtsrify-j57s8": write tcp [2601:8c:4881:8100:dc22:26c1:6182:aae0]:58275->[2606:4700::6812:85a]:443: wsasend: An existing connection was forcibly closed by the remote host.)
2023-09-16 23:26:56 DEBUG : pacer: Rate limited, increasing sleep to 10ms
2023-09-16 23:27:13 DEBUG : pacer: Reducing sleep to 0s
2023-09-16 23:27:13 DEBUG : nasa.pmtiles: multi-thread copy: chunk 1/4 failed: multi-thread copy: failed to write chunk: failed to upload chunk 1 with 1073741824 bytes: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.
        status code: 403, request id: , host id:
2023-09-16 23:27:13 DEBUG : nasa.pmtiles: multi-thread copy: chunk 2/4 (1073741824-2147483648) size 1Gi starting
2023-09-16 23:27:16 DEBUG : nasa.pmtiles: multi-thread copy: chunk 2/4 failed: multi-thread copy: failed to write chunk: failed to upload chunk 2 with 1073741824 bytes: RequestCanceled: request context canceled
caused by: context canceled
2023-09-16 23:27:16 DEBUG : nasa.pmtiles: multi-thread copy: cancelling transfer on exit
2023-09-16 23:27:16 DEBUG : nasa.pmtiles: multipart upload "AJYdKGi5ZfdVpmtktRqXke34KURxUy8Rn7iacnfPQ4s5Z_MdpNeM8jzD9S4FheM0YLGrBYROqBQju5iyzkxSdSjvuH1xSBBK2FiQUsTtUSIAgN2-C2igcM9kJbhhSKtpq-9VXuj-qRK6S9MlLkcl2LCxBh9kfraeo4UnelEEo9GyrXjNS7UvM38vvkAZg_PHiTIRqWW2CJgNwJctQ9-0WbTX273P_br4iBllCYfSoFonDI5J6xawdztL-TYZoRGMAy_DKhkudeWFk7C43eFWW078HJ2iwUPF0-6x2tDU8_MumHEkkM5B8A5fyECvNwZGuGNcvTIODhTtsrify-j57s8" aborted
2023-09-16 23:27:16 ERROR : nasa.pmtiles: Failed to copy: multi-thread copy: failed to write chunk: failed to upload chunk 1 with 1073741824 bytes: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.
        status code: 403, request id: , host id:
2023-09-16 23:27:16 ERROR : Attempt 3/3 failed with 1 errors and: multi-thread copy: failed to write chunk: failed to upload chunk 1 with 1073741824 bytes: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.
        status code: 403, request id: , host id:
Transferred:        1.181 GiB / 1.181 GiB, 100%, 1.730 MiB/s, ETA 0s
Errors:                 1 (retrying may help)
Elapsed time:       2m0.1s
2023/09/16 23:27:16 INFO  :
Transferred:        1.181 GiB / 1.181 GiB, 100%, 1.730 MiB/s, ETA 0s
Errors:                 1 (retrying may help)
Elapsed time:       2m0.1s

2023/09/16 23:27:16 DEBUG : 6 go routines active
2023/09/16 23:27:16 Failed to copy: multi-thread copy: failed to write chunk: failed to upload chunk 1 with 1073741824 bytes: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.
        status code: 403, request id: , host id:

What is the issue? It is ridiculous that Cloudflare has no alternative to uploading large files than r2 (other than uploading the whole thing to AWS S3 first and then slurping it, at some cost), and it clearly doesn't work that well.

Is there any update on this issue? I am at my wit's end here. The consensus seems to be that this is the (confusing, unhelpful) way that r2 signals that you are pinging them too quickly or something. I see no way to actually reduce things so that it works. These files are not immeasurably large; I would gladly wait days for them to upload if that would make it happen. As it is, I cannot upload them at all.

Here's one where I tried to increase the delay between API calls dramatically. Note that it fails WHILE UPLOADING THE CHUNK -- it aborts it. Something is seriously wrong here.

D:\maptiles>rclone copy nasa.pmtiles r2:osmtiles/ --s3-disable-checksum --s3-chunk-size=1G --s3-upload-cutoff=5G --s3-upload-concurrency=1 --drive-pacer-burst=1 --drive-pacer-min-sleep=300ms --progress -vv
2023/09/16 23:31:11 DEBUG : rclone: Version "v1.64.0" starting with parameters ["rclone" "copy" "nasa.pmtiles" "r2:osmtiles/" "--s3-disable-checksum" "--s3-chunk-size=1G" "--s3-upload-cutoff=5G" "--s3-upload-concurrency=1" "--drive-pacer-burst=1" "--drive-pacer-min-sleep=300ms" "--progress" "-vv"]
2023/09/16 23:31:11 DEBUG : Creating backend with remote "nasa.pmtiles"
2023/09/16 23:31:11 DEBUG : Using config file from "C:\\Users\\User\\AppData\\Roaming\\rclone\\rclone.conf"
2023/09/16 23:31:11 DEBUG : fs cache: adding new entry for parent of "nasa.pmtiles", "//?/D:/maptiles"
2023/09/16 23:31:11 DEBUG : Creating backend with remote "r2:osmtiles/"
2023/09/16 23:31:11 DEBUG : r2: detected overridden config - adding "{F7-Ct}" suffix to name
2023/09/16 23:31:11 DEBUG : Resolving service "s3" region "auto"
2023/09/16 23:31:11 DEBUG : fs cache: renaming cache item "r2:osmtiles/" to be canonical "r2{F7-Ct}:osmtiles"
2023-09-16 23:31:11 DEBUG : nasa.pmtiles: Sizes differ (src 4060292083 vs dst 52428800)
2023-09-16 23:31:17 DEBUG : nasa.pmtiles: open chunk writer: started multipart upload: AMmyxgJRPX4Uou7OvPyWP0Z-rn0fdb6621vffz7HMfoSC7OMjLe6eGeiNMn6lUTSWt_Yq3aw9KgQPO6IkDR8_aMOxc7wok_fouthGhzRdbNaNxR4EKlaAuDDU06hzmAb3devERXPPrnmvkLCtS2pugBtok8SLnQ36fw5-oDho9yF4FtXuB4iROlZS0KCVHcqGtbaKXAEaIBUwhrKJLVRb62zc1ABWVd0m-C65F20-sJtTnt_4H3qPBnXmvVQ3o5jj9RT7RtAGLmzAb3raGGl3jK5KEyu_ZVrivr3Td4akUTGs9kJ0oSGMDg-GQn7yxLLp1ZUTUILVmn0nUim1S0vtMA
2023-09-16 23:31:17 DEBUG : nasa.pmtiles: multi-thread copy: using backend concurrency of 1 instead of --multi-thread-streams 4
2023-09-16 23:31:17 DEBUG : nasa.pmtiles: Starting multi-thread copy with 4 chunks of size 1Gi with 1 parallel streams
2023-09-16 23:31:17 DEBUG : nasa.pmtiles: multi-thread copy: chunk 1/4 (0-1073741824) size 1Gi starting
2023-09-16 23:32:36 DEBUG : nasa.pmtiles: multi-thread copy: chunk 1/4 failed: multi-thread copy: failed to write chunk: failed to upload chunk 1 with 1073741824 bytes: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.
        status code: 403, request id: , host id:
2023-09-16 23:32:36 DEBUG : nasa.pmtiles: multi-thread copy: chunk 2/4 (1073741824-2147483648) size 1Gi starting
2023-09-16 23:32:39 DEBUG : nasa.pmtiles: multi-thread copy: chunk 2/4 failed: multi-thread copy: failed to write chunk: failed to upload chunk 2 with 1073741824 bytes: RequestCanceled: request context canceled
caused by: context canceled
2023-09-16 23:32:39 DEBUG : nasa.pmtiles: multi-thread copy: cancelling transfer on exit
2023-09-16 23:32:39 DEBUG : nasa.pmtiles: multipart upload "AMmyxgJRPX4Uou7OvPyWP0Z-rn0fdb6621vffz7HMfoSC7OMjLe6eGeiNMn6lUTSWt_Yq3aw9KgQPO6IkDR8_aMOxc7wok_fouthGhzRdbNaNxR4EKlaAuDDU06hzmAb3devERXPPrnmvkLCtS2pugBtok8SLnQ36fw5-oDho9yF4FtXuB4iROlZS0KCVHcqGtbaKXAEaIBUwhrKJLVRb62zc1ABWVd0m-C65F20-sJtTnt_4H3qPBnXmvVQ3o5jj9RT7RtAGLmzAb3raGGl3jK5KEyu_ZVrivr3Td4akUTGs9kJ0oSGMDg-GQn7yxLLp1ZUTUILVmn0nUim1S0vtMA" aborted
2023-09-16 23:32:39 ERROR : nasa.pmtiles: Failed to copy: multi-thread copy: failed to write chunk: failed to upload chunk 1 with 1073741824 bytes: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.
        status code: 403, request id: , host id:
2023-09-16 23:32:39 ERROR : Attempt 1/3 failed with 1 errors and: multi-thread copy: failed to write chunk: failed to upload chunk 1 with 1073741824 bytes: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.
        status code: 403, request id: , host id:
2023-09-16 23:32:39 DEBUG : nasa.pmtiles: Sizes differ (src 4060292083 vs dst 52428800)
2023-09-16 23:32:45 DEBUG : nasa.pmtiles: open chunk writer: started multipart upload: ANCTP6erKY5_PW1NoXUp1LjfG6c3M39qUAh4Z_dWW1QC_PDex_819Z3iMET-sZ_RGaMCR01NvoR-Fxmpw_3XFGGTEm6EHAAhKyEqdll1F2dfdLl1rdbEyzpXyAPFH7vhI8FPZvtxfX83lPMBEvgJRIySpPNFQZQj1tPjLI-102EHQT3qU071_TagH4oMk8K2RW-UWsnC62CVcrS0L25doVtdAIJ46nxAv51Yeysmg0Mb7JrPF3SBc5tYCMbDTlAoduFO6dcEVOuUlV1pFe6oVjUG7kqtfQ1f5cvCymPClSWKV4QO3fHNasgdPh3iyA0kKIdesx9pIAuMj_TIODVH2t0
2023-09-16 23:32:45 DEBUG : nasa.pmtiles: multi-thread copy: using backend concurrency of 1 instead of --multi-thread-streams 4
2023-09-16 23:32:45 DEBUG : nasa.pmtiles: Starting multi-thread copy with 4 chunks of size 1Gi with 1 parallel streams
2023-09-16 23:32:45 DEBUG : nasa.pmtiles: multi-thread copy: chunk 1/4 (0-1073741824) size 1Gi starting
2023-09-16 23:33:37 DEBUG : nasa.pmtiles: multi-thread copy: chunk 1/4 failed: multi-thread copy: failed to write chunk: failed to upload chunk 1 with 1073741824 bytes: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.
        status code: 403, request id: , host id:
2023-09-16 23:33:37 DEBUG : nasa.pmtiles: multi-thread copy: chunk 2/4 (1073741824-2147483648) size 1Gi starting
2023-09-16 23:33:40 DEBUG : nasa.pmtiles: multi-thread copy: chunk 2/4 failed: multi-thread copy: failed to write chunk: failed to upload chunk 2 with 1073741824 bytes: RequestCanceled: request context canceled
caused by: context canceled
2023-09-16 23:33:40 DEBUG : nasa.pmtiles: multi-thread copy: cancelling transfer on exit
2023-09-16 23:33:40 DEBUG : nasa.pmtiles: multipart upload "ANCTP6erKY5_PW1NoXUp1LjfG6c3M39qUAh4Z_dWW1QC_PDex_819Z3iMET-sZ_RGaMCR01NvoR-Fxmpw_3XFGGTEm6EHAAhKyEqdll1F2dfdLl1rdbEyzpXyAPFH7vhI8FPZvtxfX83lPMBEvgJRIySpPNFQZQj1tPjLI-102EHQT3qU071_TagH4oMk8K2RW-UWsnC62CVcrS0L25doVtdAIJ46nxAv51Yeysmg0Mb7JrPF3SBc5tYCMbDTlAoduFO6dcEVOuUlV1pFe6oVjUG7kqtfQ1f5cvCymPClSWKV4QO3fHNasgdPh3iyA0kKIdesx9pIAuMj_TIODVH2t0" aborted
2023-09-16 23:33:40 ERROR : nasa.pmtiles: Failed to copy: multi-thread copy: failed to write chunk: failed to upload chunk 1 with 1073741824 bytes: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.
        status code: 403, request id: , host id:
2023-09-16 23:33:40 ERROR : Attempt 2/3 failed with 1 errors and: multi-thread copy: failed to write chunk: failed to upload chunk 1 with 1073741824 bytes: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.
        status code: 403, request id: , host id:
2023-09-16 23:33:40 DEBUG : nasa.pmtiles: Sizes differ (src 4060292083 vs dst 52428800)
2023-09-16 23:33:46 DEBUG : nasa.pmtiles: open chunk writer: started multipart upload: AJdV13lW3LYY7cPAXh40o2roqa0K6ZHZ7FLLi6MwKf-koM1jBdO1vtUOyn9OnKZc7BVTOUyt610_f2VV5xGxRQFU8oS3G6SoshyiyngIfXoTx-boZOIdCrF7ZIs0cQ9qZpoqEdYFp71GqnuLUtELQni0aC50BXVBm5U9P6MXtHKIqErBH2bx4qfTE3FGTO0eXjydRUs6J48JILDBFJTR_Q8nFmSHd19Q88Gm5kRgCq7YYWEIkBfSplJGJlaT440kRFXztRu0LqTHfySBbycq5e3yoVQh6FvkB3mAGA5eSpshTMxTTFj9jgw0CHW46rQ30mEoyvRrjEls6aDBQTG1bk4
2023-09-16 23:33:46 DEBUG : nasa.pmtiles: multi-thread copy: using backend concurrency of 1 instead of --multi-thread-streams 4
2023-09-16 23:33:46 DEBUG : nasa.pmtiles: Starting multi-thread copy with 4 chunks of size 1Gi with 1 parallel streams
2023-09-16 23:33:46 DEBUG : nasa.pmtiles: multi-thread copy: chunk 1/4 (0-1073741824) size 1Gi starting
2023-09-16 23:33:53 DEBUG : nasa.pmtiles: multi-thread copy: chunk 1/4 failed: multi-thread copy: failed to write chunk: failed to upload chunk 1 with 1073741824 bytes: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.
        status code: 403, request id: , host id:
2023-09-16 23:33:53 DEBUG : nasa.pmtiles: multi-thread copy: chunk 2/4 (1073741824-2147483648) size 1Gi starting
2023-09-16 23:33:56 DEBUG : nasa.pmtiles: multi-thread copy: chunk 2/4 failed: multi-thread copy: failed to write chunk: failed to upload chunk 2 with 1073741824 bytes: RequestCanceled: request context canceled
caused by: context canceled
2023-09-16 23:33:56 DEBUG : nasa.pmtiles: multi-thread copy: cancelling transfer on exit
2023-09-16 23:33:56 DEBUG : nasa.pmtiles: multipart upload "AJdV13lW3LYY7cPAXh40o2roqa0K6ZHZ7FLLi6MwKf-koM1jBdO1vtUOyn9OnKZc7BVTOUyt610_f2VV5xGxRQFU8oS3G6SoshyiyngIfXoTx-boZOIdCrF7ZIs0cQ9qZpoqEdYFp71GqnuLUtELQni0aC50BXVBm5U9P6MXtHKIqErBH2bx4qfTE3FGTO0eXjydRUs6J48JILDBFJTR_Q8nFmSHd19Q88Gm5kRgCq7YYWEIkBfSplJGJlaT440kRFXztRu0LqTHfySBbycq5e3yoVQh6FvkB3mAGA5eSpshTMxTTFj9jgw0CHW46rQ30mEoyvRrjEls6aDBQTG1bk4" aborted
2023-09-16 23:33:56 ERROR : nasa.pmtiles: Failed to copy: multi-thread copy: failed to write chunk: failed to upload chunk 1 with 1073741824 bytes: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.
        status code: 403, request id: , host id:
2023-09-16 23:33:56 ERROR : Attempt 3/3 failed with 1 errors and: multi-thread copy: failed to write chunk: failed to upload chunk 1 with 1073741824 bytes: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.
        status code: 403, request id: , host id:
Transferred:      342.344 MiB / 342.344 MiB, 100%, 2.167 MiB/s, ETA 0s
Errors:                 1 (retrying may help)
Elapsed time:      2m45.4s
2023/09/16 23:33:56 INFO  :
Transferred:      342.344 MiB / 342.344 MiB, 100%, 2.167 MiB/s, ETA 0s
Errors:                 1 (retrying may help)
Elapsed time:      2m45.4s

2023/09/16 23:33:56 DEBUG : 6 go routines active
2023/09/16 23:33:56 Failed to copy: multi-thread copy: failed to write chunk: failed to upload chunk 1 with 1073741824 bytes: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.
        status code: 403, request id: , host id:

Any advice you can give is appreciated. This is driving me nuts...! Thank you.

ncw · September 17, 2023, 8:33am

This looks like the signature of individual chunks is being rejected.

That is a strange problem.

This tends to indicate a problem with your access and secret key but I'm not sure what. Maybe you put them into the config in a different case or maybe there are spaces. That's unlikely but I'd remake the config from scratch with rclone config to check.

Can you reproduce this with a file full of zeros?

You can create one with

rclone test makefile --zero 4G 4gzeroes

If that reproduces the problem then run the transfer with -vv --dump bodies --log-file rclone.log

Compress the log file and post a link to it here (or send me nick@craig-wood.com it directly with a link to this forum page for context) and I'll take a look to see if I can work out what is happening.

Thanks

vladpavlovski · September 24, 2023, 6:19pm

Thank you for your suggestions. I encountered the same problem and resolved it by removing spaces before and after the equal sign in rclone.conf file.

nucleon · October 2, 2023, 9:14pm

Thanks for taking the time to respond. I'm sorry about my taking time to get back -- I had to put this on the back burner for a couple of weeks.

So I tried it with other files, and even with a whole new bucket configuration and API key, and even on a totally different computer with a different OS (Mac), and got exactly the same problem.

BUT then I had the random insight to try using a hardwired ethernet connection into my router, instead of sending it over WiFi, and... that seems to have resolved the problem! One of my files (the 4GB one) has uploaded without issue, and the 100GB one seems to be uploading fine.

Which strikes me as bizarre and insane. I don't know enough about networking to have any clue why that would be a different situation in this case. My only theory is that perhaps my router was doing something dumb like rotating my the IP address every so many minutes, or changing how it was sending the packets, and this made the authentication fail? I don't know.

Anyway, I hope some future frustrated Googler finds this useful!

nucleon · October 3, 2023, 8:57pm

Well, I spoke to soon. I did still end up having issues with the 100 GB file, just more infrequently. The errors were both the SignatureDoesNotMatch error as well as clear issues of rate limiting, as seen in this sample from a failed session:

2023-10-02 22:12:02 DEBUG : base.pmtiles: multi-thread copy: chunk 370/1051 (38692454400-38797312000) size 100Mi starting
2023-10-02 22:12:43 DEBUG : pacer: low level retry 1/2 (error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.
        status code: 403, request id: , host id: )
2023-10-02 22:12:43 DEBUG : pacer: Rate limited, increasing sleep to 10ms
2023-10-02 22:13:17 DEBUG : pacer: low level retry 2/2 (error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.
        status code: 403, request id: , host id: )
2023-10-02 22:13:17 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2023-10-02 22:13:17 DEBUG : base.pmtiles: multi-thread copy: chunk 370/1051 failed: multi-thread copy: failed to write chunk: failed to upload chunk 370 with 104857600 bytes: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.
        status code: 403, request id: , host id:

On the theory that maybe there was still some sort of IP-hopping going on, I tried using it with a VPN (which I thought might guarantee a static IP address, but what do I know about that), and that had no impact (other than decreasing my upload speed).

The settings I used that eventually got me almost-success were:

rclone copy base.pmtiles r2:maps/ --s3-chunk-size=300M --s3-upload-concurrency=1 --retries 1000 --low-level-retries 1000  --progress -vv

Which still triggered some errors, but seemed like it was working. I ran it for almost 18 hours... and then it crapped out at the end and appeared to restart:

2023-10-03 14:33:16 DEBUG : base.pmtiles: multi-thread copy: chunk 339/351 (106325606400-106640179200) size 300Mi starting
2023-10-03 14:33:35 DEBUG : pacer: low level retry 1/2 (error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.
        status code: 403, request id: , host id: )
2023-10-03 14:33:35 DEBUG : pacer: Rate limited, increasing sleep to 10ms
2023-10-03 14:35:29 DEBUG : pacer: low level retry 2/2 (error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.
        status code: 403, request id: , host id: )
2023-10-03 14:35:29 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2023-10-03 14:35:29 DEBUG : base.pmtiles: multi-thread copy: chunk 339/351 failed: multi-thread copy: failed to write chunk: failed to upload chunk 339 with 314572800 bytes: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.
        status code: 403, request id: , host id:
2023-10-03 14:35:29 DEBUG : base.pmtiles: multi-thread copy: chunk 340/351 (106640179200-106954752000) size 300Mi starting
2023-10-03 14:35:32 DEBUG : pacer: low level retry 1/2 (error RequestCanceled: request context canceled
caused by: context canceled)
2023-10-03 14:35:32 DEBUG : pacer: Rate limited, increasing sleep to 40ms
2023-10-03 14:35:32 DEBUG : pacer: low level retry 2/2 (error BodyHashError: failed to compute body hashes
caused by: context canceled)
2023-10-03 14:35:32 DEBUG : pacer: Rate limited, increasing sleep to 80ms
2023-10-03 14:35:32 DEBUG : base.pmtiles: multi-thread copy: chunk 340/351 failed: multi-thread copy: failed to write chunk: failed to upload chunk 340 with 314572800 bytes: BodyHashError: failed to compute body hashes
caused by: context canceled
2023-10-03 14:35:33 DEBUG : pacer: Reducing sleep to 60ms
2023-10-03 14:35:33 DEBUG : base.pmtiles: multipart upload "AJipXZmbvZ83fEF21soyu-ZYAB0Xiia0hrdBoTPfipi_bNPrOIjYW9Q-2p2pypIQPsklOn6Za9x9x2GOCGhqXq_dX2OZ5nS69HPX7C9jqeY4wCo0JTQB0MBzT8oWGNNwH7_SA-i4i71ma-5w188RDDA9_gF-xqPOU-lKlkkSlugJPtNAaczuTmtRO_Z0VQqfK8q1XuRAsk8WaJ3TW5I1yw-Jbu-6bYXKCOUivWHWX5AFTFpQvimCM6PxOHGaKi9Evg5XSeSzKlPp895ZrkF_q3HVQGscqcGQ1sVlPoO9IWa1SUK_wcPAMJnbeNBpcc5zb1d7Yd8LPa7PNRZOdPkRkys" finished
2023-10-03 14:35:33 DEBUG : base.pmtiles: multi-thread copy: cancelling transfer on exit
2023-10-03 14:35:33 DEBUG : pacer: Reducing sleep to 45ms
2023-10-03 14:35:33 DEBUG : base.pmtiles: multipart upload "AJipXZmbvZ83fEF21soyu-ZYAB0Xiia0hrdBoTPfipi_bNPrOIjYW9Q-2p2pypIQPsklOn6Za9x9x2GOCGhqXq_dX2OZ5nS69HPX7C9jqeY4wCo0JTQB0MBzT8oWGNNwH7_SA-i4i71ma-5w188RDDA9_gF-xqPOU-lKlkkSlugJPtNAaczuTmtRO_Z0VQqfK8q1XuRAsk8WaJ3TW5I1yw-Jbu-6bYXKCOUivWHWX5AFTFpQvimCM6PxOHGaKi9Evg5XSeSzKlPp895ZrkF_q3HVQGscqcGQ1sVlPoO9IWa1SUK_wcPAMJnbeNBpcc5zb1d7Yd8LPa7PNRZOdPkRkys" aborted
2023-10-03 14:35:33 DEBUG : base.pmtiles: Received error: multi-thread copy: failed to write chunk: failed to upload chunk 339 with 314572800 bytes: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.
        status code: 403, request id: , host id:  - low level retry 2/1000
2023-10-03 14:35:34 DEBUG : pacer: Reducing sleep to 33.75ms
2023-10-03 14:35:34 DEBUG : base.pmtiles: open chunk writer: started multipart upload: APuBh858tq3izxvK4zO5vPoHvgB2nCgoHruW9Fu0wDEx3TqjEh26Rist_ZTwgXblRCggXCY5n6DEUt7w7oNxcKuNqKN8Sk9-h850YUWLZek9kJTtvXXsoAhaGxZzKx5pQEYO2wY8K4tKCaIPhVE1iC8S_8iwXorUzMCzu9iXzDyaN9ZSIcI3YAYfkJizAjc2XOHrPTdZrhVp2TsuqCWENRsHPmJNHPGMpz_NQEp3Y5akxz-1KAIaETGFYXWs7bGnmgwJKArE_3cX6pAQm16x9gnbUI6N-OK_AagBD3NQcRwG-DbEW8C8m-6hr26zGLYg_fKRBTI_sKA8hI-I3A2IiMI
2023-10-03 14:35:34 DEBUG : base.pmtiles: multi-thread copy: using backend concurrency of 1 instead of --multi-thread-streams 4
2023-10-03 14:35:34 DEBUG : base.pmtiles: Starting multi-thread copy with 351 chunks of size 300Mi with 1 parallel streams
2023-10-03 14:35:34 DEBUG : base.pmtiles: multi-thread copy: chunk 1/351 (0-314572800) size 300Mi starting

And then it gave me an ETA of +3 weeks (the upload speed dropped to nothing -- 44 KB/s down from 2-3 MB/s) so I stopped it.

Altogether it uploaded 248GiB of the file (of a 100 GiB file!), claimed it had somehow made it to 66% of the total.

Also, I tried the "no spaces in the config file" suggestion above and it had no impact that I could discern.

I'm not sure what to try next. I might give it one more go tonight with settings like the above (maybe it was a momentary fluke in my network or something). Very frustrating. If I can't get this to work I guess I have to abandon rclone and try another way (like eating the AWS S3 upload cost and then trying to migrate it to R2 from there -- though for files >50GB I'm not sure that's even possible).

ncw · October 4, 2023, 2:39pm

This is very strange. The fact that it works some of the time and not others.

That means that rclone thought it had uploaded 66% of the chunks successfully (66GB), but on average it had try 248/66 = 3.75 times for each one!

That means there is something very wrong somewhere!

Ideas as for what

Some sort of networking problem (eg silently corrupting packets - update drivers etc)
A bug in R2 (check the r2 discord)
Memory error in your computer (check with memtest86)
A bug in rclone

I put the list in order with what I consider to be the most likely first!

Have you got another computer you could try this transfer from? That would eliminate some of the above.

nucleon · October 7, 2023, 2:39am

So I put the file on an external drive, took it to work, and set it up to upload there AND IT WORKED. Which suggests it was something to do with either my home network or my ISP or something. Which is annoying but at least an answer.

ncw · October 7, 2023, 8:27am

Ok that is a good step.

It could be your computer or network. Have you got a different computer you could try at home?

system · November 6, 2023, 8:28am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.