Cloudflare R2 -> R2 bucket copy

Help and Support
1

What is the problem you are having with rclone?

The use case is setting up an official mirror for an open-source project which hosts elevation data.
Currently the dataset is 800 GB, but in the future it'll grow into 10 TB.

The problem: I'd like to copy a single file bigger than available local disk, from one public R2 bucket and upload to another R2 bucket.

I don't have access to the source bucket via API, as that belongs to the upstream project. Because of this, the source is not technically a bucket in rclone sense, but a public bucket "mounted" to a production subdomain. It supports HTTP Range requests.

Would this be possible via rclone? I mean downloading supports HTTP Range requests and uploading supports multi-part splitting, so it should be simple to connect these two.

I'd like to know if rclone supports this, or I'd need to code this myself.

Which cloud storage system are you using? (eg Google Drive)

Cloudflare R2

2

not sure what that means?
how do you access the bucket? what copy tools are you currently using?

3

So the upstream project uploads file to their bucket, maybe via rclone.

In Cloudflare they give a public subdomain for that bucket. Exactly like in AWS S3.

Cloudflare Dashboard calls this "Custom Domain", it looks like this:

CleanShot 2025-10-21 at 23.38.43@2x
CleanShot 2025-10-21 at 23.38.43@2x1414×366 31.7 KB

Then I download a single file from that custom domain, which I'd like to upload to my own bucket, via rclone.
I download via curl/wget/aria2.

4

have you tried a rclone http remote or rclone copyurl

5

I haven't. Does it support range request -> matching multi-part automatically?

6

I think I'll need to use

rclone copy :http:

but I'm still not sure how to get this "matching" range requests working.

7

start with a simple command.
rclone ls --http-url=https://example.com :http: -vv

no idea.
fwiw, maybe run the rclone copy -vv command, see what happens?

not sure what that means? but rclone has chunker

8

OK, here are the commands:

export RCLONE_CONFIG=/dev/null

Listing is not possible, I guess that's expected. I only want to download individual files.

rclone ls --http-url=https://download.mapterhorn.com/ :http: -vv
2025/10/23 11:14:46 DEBUG : Setting --config "/dev/null" from environment variable RCLONE_CONFIG="/dev/null"
2025/10/23 11:14:46 DEBUG : rclone: Version "v1.71.2" starting with parameters ["rclone" "ls" "--http-url=https://download.mapterhorn.com/" ":http:" "-vv"]
2025/10/23 11:14:46 DEBUG : Creating backend with remote ":http:"
2025/10/23 11:14:46 DEBUG : Using config file from ""
2025/10/23 11:14:46 DEBUG : :http: detected overridden config - adding "{GawpC}" suffix to name
2025/10/23 11:14:46 DEBUG : Root: https://download.mapterhorn.com/
2025/10/23 11:14:46 DEBUG : fs cache: renaming cache item ":http:" to be canonical ":http{GawpC}:"
2025/10/23 11:14:46 ERROR : error listing: error listing "": directory not found
2025/10/23 11:14:46 DEBUG : 6 go routines active
2025/10/23 11:14:46 NOTICE: Failed to ls with 2 errors: last error was: error listing "": directory not found

copyurl:

rclone copyurl https://download.mapterhorn.com/planet.pmtiles out --dump headers
2025/10/23 11:18:19 NOTICE: Automatically setting -vv as --dump is enabled
2025/10/23 11:18:19 DEBUG : rclone: Version "v1.71.2" starting with parameters ["rclone" "copyurl" "https://download.mapterhorn.com/planet.pmtiles" "out" "--dump" "headers"]
2025/10/23 11:18:19 DEBUG : Creating backend with remote "."
2025/10/23 11:18:19 DEBUG : Using config file from ""
2025/10/23 11:18:19 DEBUG : fs cache: renaming cache item "." to be canonical "/Users/user/Downloads"
2025/10/23 11:18:19 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2025/10/23 11:18:19 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/10/23 11:18:19 DEBUG : HTTP REQUEST (req 0x140003bc000)
2025/10/23 11:18:19 DEBUG : GET /planet.pmtiles HTTP/1.1
Host: download.mapterhorn.com
User-Agent: rclone/v1.71.2
Accept-Encoding: gzip

2025/10/23 11:18:19 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/10/23 11:18:19 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/10/23 11:18:19 DEBUG : HTTP RESPONSE (req 0x140003bc000)
2025/10/23 11:18:19 DEBUG : HTTP/2.0 200 OK
Content-Length: 572674039099
Accept-Ranges: bytes
Cf-Cache-Status: DYNAMIC
Cf-Ray: 9930257bb8dcc1cd-BUD
Content-Type: application/octet-stream
Date: Thu, 23 Oct 2025 09:18:19 GMT
Etag: "22ee711f68ab7319fbe42a9c08940e9a-10000"
Last-Modified: Sun, 19 Oct 2025 22:19:25 GMT
Nel: {"report_to":"cf-nel","success_fraction":0.0,"max_age":604800}
Report-To: {"group":"cf-nel","max_age":604800,"endpoints":[{"url":"https://a.nel.cloudflare.com/report/v4?s=9shRtu24BJjVxIqvpy7Jxm6mbiU9NdfuLRMyBxTSVLgKjOJFpnGXfLuChxKEy%2BPlCdwLSc4lYQ22HxCDT3afxcpqy4xzujajTcQfTv5h%2BEC4RVB3"}]}
Server: cloudflare
Vary: Accept-Encoding

2025/10/23 11:18:19 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

--multi-thread doesn't have any effect on copyurl

zsmac16 user ~/Downloads $ rclone copyurl https://download.mapterhorn.com/planet.pmtiles out --dump headers --multi-thread-streams 8 --multi-thread-cutoff 100M -vv
2025/10/23 11:22:56 DEBUG : Setting --config "/dev/null" from environment variable RCLONE_CONFIG="/dev/null"
2025/10/23 11:22:56 NOTICE: Automatically setting -vv as --dump is enabled
2025/10/23 11:22:56 DEBUG : rclone: Version "v1.71.2" starting with parameters ["rclone" "copyurl" "https://download.mapterhorn.com/planet.pmtiles" "out" "--dump" "headers" "--multi-thread-streams" "8" "--multi-thread-cutoff" "100M" "-vv"]
2025/10/23 11:22:56 DEBUG : Creating backend with remote "."
2025/10/23 11:22:56 DEBUG : Using config file from ""
2025/10/23 11:22:56 DEBUG : fs cache: renaming cache item "." to be canonical "/Users/user/Downloads"
2025/10/23 11:22:56 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2025/10/23 11:22:56 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/10/23 11:22:56 DEBUG : HTTP REQUEST (req 0x14000836000)
2025/10/23 11:22:56 DEBUG : GET /planet.pmtiles HTTP/1.1
Host: download.mapterhorn.com
User-Agent: rclone/v1.71.2
Accept-Encoding: gzip

2025/10/23 11:22:56 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/10/23 11:22:56 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/10/23 11:22:56 DEBUG : HTTP RESPONSE (req 0x14000836000)
2025/10/23 11:22:56 DEBUG : HTTP/2.0 200 OK
Content-Length: 572674039099
Accept-Ranges: bytes
Cf-Cache-Status: DYNAMIC
Cf-Ray: 99302c3ee90c68b6-BUD
Content-Type: application/octet-stream
Date: Thu, 23 Oct 2025 09:22:56 GMT
Etag: "22ee711f68ab7319fbe42a9c08940e9a-10000"
Last-Modified: Sun, 19 Oct 2025 22:19:25 GMT
Nel: {"report_to":"cf-nel","success_fraction":0.0,"max_age":604800}
Report-To: {"group":"cf-nel","max_age":604800,"endpoints":[{"url":"https://a.nel.cloudflare.com/report/v4?s=gs5NxrGnRHVfgMmVxu4LtWo6xLLgja3gF%2FcY%2FsfExuBwYrtyrO23TRa13UK6laXIQs8QABbPM5Yt8dAuELYkNvzKiPizDUJt6mKv1MSfk1wMemj4"}]}
Server: cloudflare
Vary: Accept-Encoding

2025/10/23 11:22:56 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
^C

copy to local disk seem to be using range requests. now the question is if this is synced to the upload multipart requests.

zsmac16 user ~/Downloads $ rclone copy ":http,url='https://download.mapterhorn.com':/planet.pmtiles" . --dump headers -vv
2025/10/23 11:29:24 DEBUG : Setting --config "/dev/null" from environment variable RCLONE_CONFIG="/dev/null"
2025/10/23 11:29:24 NOTICE: Automatically setting -vv as --dump is enabled
2025/10/23 11:29:24 DEBUG : rclone: Version "v1.71.2" starting with parameters ["rclone" "copy" ":http,url='https://download.mapterhorn.com':/planet.pmtiles" "." "--dump" "headers" "-vv"]
2025/10/23 11:29:24 DEBUG : Creating backend with remote ":http,url='https://download.mapterhorn.com':/planet.pmtiles"
2025/10/23 11:29:24 DEBUG : Using config file from ""
2025/10/23 11:29:24 DEBUG : :http: detected overridden config - adding "{QQNBx}" suffix to name
2025/10/23 11:29:24 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2025/10/23 11:29:24 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/10/23 11:29:24 DEBUG : HTTP REQUEST (req 0x140008303c0)
2025/10/23 11:29:24 DEBUG : HEAD /planet.pmtiles HTTP/1.1
Host: download.mapterhorn.com
User-Agent: rclone/v1.71.2

2025/10/23 11:29:24 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/10/23 11:29:24 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/10/23 11:29:24 DEBUG : HTTP RESPONSE (req 0x140008303c0)
2025/10/23 11:29:24 DEBUG : HTTP/2.0 200 OK
Content-Length: 572674039099
Accept-Ranges: bytes
Cf-Cache-Status: DYNAMIC
Cf-Ray: 993035b72a48fa35-BUD
Content-Type: application/octet-stream
Date: Thu, 23 Oct 2025 09:29:24 GMT
Etag: "22ee711f68ab7319fbe42a9c08940e9a-10000"
Last-Modified: Sun, 19 Oct 2025 22:19:25 GMT
Nel: {"report_to":"cf-nel","success_fraction":0.0,"max_age":604800}
Report-To: {"group":"cf-nel","max_age":604800,"endpoints":[{"url":"https://a.nel.cloudflare.com/report/v4?s=ndeTMKc3HcyWC0bN03tGq8r1T%2FOGMYFIrEU7Bnli7S5y1wBlDE5epK23ldlZzCjp9q%2FQk8HV1TZpwqC3GR2UTH8x0rVTlEADK9kHCkA84k6tqVUq"}]}
Server: cloudflare
Vary: Accept-Encoding

2025/10/23 11:29:24 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/10/23 11:29:24 DEBUG : Assuming path is a file as HEAD response is success (200 OK)
2025/10/23 11:29:24 DEBUG : If path is a directory you must add a trailing '/'
2025/10/23 11:29:24 DEBUG : Root: https://download.mapterhorn.com/
2025/10/23 11:29:24 DEBUG : fs cache: renaming child cache item ":http,url='https://download.mapterhorn.com':/planet.pmtiles" to be canonical for parent ":http{QQNBx}:"
2025/10/23 11:29:24 DEBUG : Creating backend with remote "."
2025/10/23 11:29:24 DEBUG : fs cache: renaming cache item "." to be canonical "/Users/user/Downloads"
2025/10/23 11:29:24 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/10/23 11:29:24 DEBUG : HTTP REQUEST (req 0x140003ba000)
2025/10/23 11:29:24 DEBUG : HEAD /planet.pmtiles HTTP/1.1
Host: download.mapterhorn.com
User-Agent: rclone/v1.71.2

2025/10/23 11:29:24 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/10/23 11:29:24 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/10/23 11:29:24 DEBUG : HTTP RESPONSE (req 0x140003ba000)
2025/10/23 11:29:24 DEBUG : HTTP/2.0 200 OK
Content-Length: 572674039099
Accept-Ranges: bytes
Cf-Cache-Status: DYNAMIC
Cf-Ray: 993035b7aba0fa35-BUD
Content-Type: application/octet-stream
Date: Thu, 23 Oct 2025 09:29:24 GMT
Etag: "22ee711f68ab7319fbe42a9c08940e9a-10000"
Last-Modified: Sun, 19 Oct 2025 22:19:25 GMT
Nel: {"report_to":"cf-nel","success_fraction":0.0,"max_age":604800}
Report-To: {"group":"cf-nel","max_age":604800,"endpoints":[{"url":"https://a.nel.cloudflare.com/report/v4?s=hhZxu0omCeCjaY%2FINKaEnQGfsk9qz70zLQ%2BF2zBH81TX7xYAsX4OLXALomeIkq3IxRSIjZ3auDlywXmj5z5E9aRj4JCh%2FYEIyNn9Xy27I4jQGlyt"}]}
Server: cloudflare
Vary: Accept-Encoding

2025/10/23 11:29:24 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/10/23 11:29:24 DEBUG : planet.pmtiles: Need to transfer - File not found at Destination
2025/10/23 11:29:24 DEBUG : planet.pmtiles: multi-thread copy: disabling buffering because destination uses OpenWriterAt
2025/10/23 11:29:24 DEBUG : planet.pmtiles: multi-thread copy: write buffer set to 131072
2025/10/23 11:29:24 DEBUG : planet.pmtiles: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 4
2025/10/23 11:29:24 DEBUG : planet.pmtiles: Starting multi-thread copy with 8534 chunks of size 64Mi with 4 parallel streams
2025/10/23 11:29:24 DEBUG : planet.pmtiles: multi-thread copy: chunk 4/8534 (201326592-268435456) size 64Mi starting
2025/10/23 11:29:24 DEBUG : planet.pmtiles: multi-thread copy: chunk 2/8534 (67108864-134217728) size 64Mi starting
2025/10/23 11:29:24 DEBUG : planet.pmtiles: multi-thread copy: chunk 3/8534 (134217728-201326592) size 64Mi starting
2025/10/23 11:29:24 DEBUG : planet.pmtiles: multi-thread copy: chunk 1/8534 (0-67108864) size 64Mi starting
2025/10/23 11:29:24 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/10/23 11:29:24 DEBUG : HTTP REQUEST (req 0x1400067e640)
2025/10/23 11:29:24 DEBUG : GET /planet.pmtiles HTTP/1.1
Host: download.mapterhorn.com
User-Agent: rclone/v1.71.2
Range: bytes=67108864-134217727

2025/10/23 11:29:24 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/10/23 11:29:24 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/10/23 11:29:24 DEBUG : HTTP REQUEST (req 0x140003badc0)
2025/10/23 11:29:24 DEBUG : GET /planet.pmtiles HTTP/1.1
Host: download.mapterhorn.com
User-Agent: rclone/v1.71.2
Range: bytes=134217728-201326591

2025/10/23 11:29:24 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/10/23 11:29:24 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/10/23 11:29:24 DEBUG : HTTP REQUEST (req 0x140003bb180)
2025/10/23 11:29:24 DEBUG : GET /planet.pmtiles HTTP/1.1
Host: download.mapterhorn.com
User-Agent: rclone/v1.71.2
Range: bytes=0-67108863

2025/10/23 11:29:24 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/10/23 11:29:24 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/10/23 11:29:24 DEBUG : HTTP REQUEST (req 0x140003bab40)
2025/10/23 11:29:24 DEBUG : GET /planet.pmtiles HTTP/1.1
Host: download.mapterhorn.com
User-Agent: rclone/v1.71.2
Range: bytes=201326592-268435455

2025/10/23 11:29:24 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/10/23 11:29:24 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/10/23 11:29:24 DEBUG : HTTP RESPONSE (req 0x140003bb180)
2025/10/23 11:29:24 DEBUG : HTTP/2.0 206 Partial Content
Content-Length: 67108864
Accept-Ranges: bytes
Cf-Cache-Status: DYNAMIC
Cf-Ray: 993035b82cc6fa35-BUD
Content-Range: bytes 0-67108863/572674039099
Content-Type: application/octet-stream
Date: Thu, 23 Oct 2025 09:29:24 GMT
Etag: "22ee711f68ab7319fbe42a9c08940e9a-10000"
Last-Modified: Sun, 19 Oct 2025 22:19:25 GMT
Nel: {"report_to":"cf-nel","success_fraction":0.0,"max_age":604800}
Report-To: {"group":"cf-nel","max_age":604800,"endpoints":[{"url":"https://a.nel.cloudflare.com/report/v4?s=NuMq0ltkz82UtQ1YG4BhAs0dWVygMR7aY9hj9KJuZdQWKsUNOXlw5zUaK%2FMPAwwYInvTqOSbgjMKpxWy6PlGQq31sy5WD90SsnWHrzmMpOBGo0D2"}]}
Server: cloudflare
Vary: Accept-Encoding

2025/10/23 11:29:24 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/10/23 11:29:24 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/10/23 11:29:24 DEBUG : HTTP RESPONSE (req 0x140003bab40)
2025/10/23 11:29:24 DEBUG : planet.pmtiles.cdabed0c.partial: writing chunk 0
2025/10/23 11:29:24 DEBUG : HTTP/2.0 206 Partial Content
Content-Length: 67108864
Accept-Ranges: bytes
Cf-Cache-Status: DYNAMIC
Cf-Ray: 993035b82cc7fa35-BUD
Content-Range: bytes 201326592-268435455/572674039099
Content-Type: application/octet-stream
Date: Thu, 23 Oct 2025 09:29:24 GMT
Etag: "22ee711f68ab7319fbe42a9c08940e9a-10000"
Last-Modified: Sun, 19 Oct 2025 22:19:25 GMT
Nel: {"report_to":"cf-nel","success_fraction":0.0,"max_age":604800}
Report-To: {"group":"cf-nel","max_age":604800,"endpoints":[{"url":"https://a.nel.cloudflare.com/report/v4?s=3nKbXA2MT94PT46N7rCWfZPEeDPyKzJJ7Q8x0mvFFlK3JMFcRPBI21LbY5tqilFiBqqkD%2BHcyde%2Fx2vOlcW2uQi08sRSvjV48ex7ni3J2Zr8AKNG"}]}
Server: cloudflare
Vary: Accept-Encoding

2025/10/23 11:29:24 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/10/23 11:29:24 DEBUG : planet.pmtiles.cdabed0c.partial: writing chunk 3
2025/10/23 11:29:24 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/10/23 11:29:24 DEBUG : HTTP RESPONSE (req 0x140003badc0)
2025/10/23 11:29:24 DEBUG : HTTP/2.0 206 Partial Content
Content-Length: 67108864
Accept-Ranges: bytes
Cf-Cache-Status: DYNAMIC
Cf-Ray: 993035b82cc5fa35-BUD
Content-Range: bytes 134217728-201326591/572674039099
Content-Type: application/octet-stream
Date: Thu, 23 Oct 2025 09:29:24 GMT
Etag: "22ee711f68ab7319fbe42a9c08940e9a-10000"
Last-Modified: Sun, 19 Oct 2025 22:19:25 GMT
Nel: {"report_to":"cf-nel","success_fraction":0.0,"max_age":604800}
Report-To: {"group":"cf-nel","max_age":604800,"endpoints":[{"url":"https://a.nel.cloudflare.com/report/v4?s=FM4nXBepLqqSkoBxawtC2dIPiZDCz4JbwxjdV014hzM5pmK%2FNJI%2F6Ujwvpxp7zP1PQMUCKCE%2FMi%2F4zhzy0h8sZ5i4PlPnHIkGerxmra5YguITWCY"}]}
Server: cloudflare
Vary: Accept-Encoding

2025/10/23 11:29:24 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/10/23 11:29:24 DEBUG : planet.pmtiles.cdabed0c.partial: writing chunk 2
2025/10/23 11:29:24 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/10/23 11:29:24 DEBUG : HTTP RESPONSE (req 0x1400067e640)
2025/10/23 11:29:24 DEBUG : HTTP/2.0 206 Partial Content
Content-Length: 67108864
Accept-Ranges: bytes
Cf-Cache-Status: DYNAMIC
Cf-Ray: 993035b82cc2fa35-BUD
Content-Range: bytes 67108864-134217727/572674039099
Content-Type: application/octet-stream
Date: Thu, 23 Oct 2025 09:29:24 GMT
Etag: "22ee711f68ab7319fbe42a9c08940e9a-10000"
Last-Modified: Sun, 19 Oct 2025 22:19:25 GMT
Nel: {"report_to":"cf-nel","success_fraction":0.0,"max_age":604800}
Report-To: {"group":"cf-nel","max_age":604800,"endpoints":[{"url":"https://a.nel.cloudflare.com/report/v4?s=snF3a5vF2G3NH8ploTwAkBTqfiXMMJ1m5YPfuPvwW%2Bv6HM24PP4%2FHN8pgi9Tb6hCQOo6wWWILp94MiFEPG6CuMKI4e1Jg%2B8wSUS5mQG9YYllcg81"}]}
Server: cloudflare
Vary: Accept-Encoding

2025/10/23 11:29:24 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/10/23 11:29:24 DEBUG : planet.pmtiles.cdabed0c.partial: writing chunk 1
2025/10/23 11:29:48 DEBUG : planet.pmtiles: multi-thread copy: chunk 1/8534 (0-67108864) size 64Mi finished
2025/10/23 11:29:48 DEBUG : planet.pmtiles: multi-thread copy: chunk 5/8534 (268435456-335544320) size 64Mi starting
2025/10/23 11:29:48 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/10/23 11:29:48 DEBUG : HTTP REQUEST (req 0x140008303c0)
2025/10/23 11:29:48 DEBUG : GET /planet.pmtiles HTTP/1.1
Host: download.mapterhorn.com
User-Agent: rclone/v1.71.2
Range: bytes=268435456-335544319

2025/10/23 11:29:48 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/10/23 11:29:48 DEBUG : planet.pmtiles: multi-thread copy: chunk 3/8534 (134217728-201326592) size 64Mi finished
2025/10/23 11:29:48 DEBUG : planet.pmtiles: multi-thread copy: chunk 2/8534 (67108864-134217728) size 64Mi finished
2025/10/23 11:29:48 DEBUG : planet.pmtiles: multi-thread copy: chunk 6/8534 (335544320-402653184) size 64Mi starting
2025/10/23 11:29:48 DEBUG : planet.pmtiles: multi-thread copy: chunk 7/8534 (402653184-469762048) size 64Mi starting
2025/10/23 11:29:48 DEBUG : planet.pmtiles: multi-thread copy: chunk 4/8534 (201326592-268435456) size 64Mi finished
2025/10/23 11:29:48 DEBUG : planet.pmtiles: multi-thread copy: chunk 8/8534 (469762048-536870912) size 64Mi starting
2025/10/23 11:29:48 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/10/23 11:29:48 DEBUG : HTTP REQUEST (req 0x1400067e000)
2025/10/23 11:29:48 DEBUG : GET /planet.pmtiles HTTP/1.1
Host: download.mapterhorn.com
User-Agent: rclone/v1.71.2
Range: bytes=402653184-469762047

2025/10/23 11:29:48 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/10/23 11:29:48 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/10/23 11:29:48 DEBUG : HTTP REQUEST (req 0x14000830780)
2025/10/23 11:29:48 DEBUG : GET /planet.pmtiles HTTP/1.1
Host: download.mapterhorn.com
User-Agent: rclone/v1.71.2
Range: bytes=335544320-402653183

2025/10/23 11:29:48 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/10/23 11:29:48 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/10/23 11:29:48 DEBUG : HTTP REQUEST (req 0x1400067e280)
2025/10/23 11:29:48 DEBUG : GET /planet.pmtiles HTTP/1.1
Host: download.mapterhorn.com
User-Agent: rclone/v1.71.2
Range: bytes=469762048-536870911

2025/10/23 11:29:48 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/10/23 11:29:48 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/10/23 11:29:48 DEBUG : HTTP RESPONSE (req 0x140008303c0)
2025/10/23 11:29:48 DEBUG : HTTP/2.0 206 Partial Content
Content-Length: 67108864
Accept-Ranges: bytes
Cf-Cache-Status: DYNAMIC
Cf-Ray: 9930364c9973fa35-BUD
Content-Range: bytes 268435456-335544319/572674039099
Content-Type: application/octet-stream
Date: Thu, 23 Oct 2025 09:29:48 GMT
Etag: "22ee711f68ab7319fbe42a9c08940e9a-10000"
Last-Modified: Sun, 19 Oct 2025 22:19:25 GMT
Nel: {"report_to":"cf-nel","success_fraction":0.0,"max_age":604800}
Report-To: {"group":"cf-nel","max_age":604800,"endpoints":[{"url":"https://a.nel.cloudflare.com/report/v4?s=cgkSdBQ6T84C%2FX9vZoAWM9N7uujWc0%2FyFf%2BHF5Yz3UQ4jpP4TDo5MOy2%2BcutVopbBfz8G0jg22IJ2vPirMCzLkMsx186btpj9KyHDxLDhlUozF4M"}]}
Server: cloudflare
Vary: Accept-Encoding

2025/10/23 11:29:48 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/10/23 11:29:48 DEBUG : planet.pmtiles.cdabed0c.partial: writing chunk 4
2025/10/23 11:29:48 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/10/23 11:29:48 DEBUG : HTTP RESPONSE (req 0x14000830780)
2025/10/23 11:29:48 DEBUG : HTTP/2.0 206 Partial Content
Content-Length: 67108864
Accept-Ranges: bytes
Cf-Cache-Status: DYNAMIC
Cf-Ray: 9930364cea6bfa35-BUD
Content-Range: bytes 335544320-402653183/572674039099
Content-Type: application/octet-stream
Date: Thu, 23 Oct 2025 09:29:48 GMT
Etag: "22ee711f68ab7319fbe42a9c08940e9a-10000"
Last-Modified: Sun, 19 Oct 2025 22:19:25 GMT
Nel: {"report_to":"cf-nel","success_fraction":0.0,"max_age":604800}
Report-To: {"group":"cf-nel","max_age":604800,"endpoints":[{"url":"https://a.nel.cloudflare.com/report/v4?s=YEkqyx%2F06lWQpDB%2FRI9mSsQqB0%2B09H7136ACIUp7%2FGabo7Lho%2BhJtkxh0wpR5yJcArxVzDMRcGvCV4fTiIMhSvbolFeYHGzy1vt9r8thOlAX40%2Fv"}]}
Server: cloudflare
Vary: Accept-Encoding

2025/10/23 11:29:48 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/10/23 11:29:48 DEBUG : planet.pmtiles.cdabed0c.partial: writing chunk 5
2025/10/23 11:29:48 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/10/23 11:29:48 DEBUG : HTTP RESPONSE (req 0x1400067e000)
2025/10/23 11:29:48 DEBUG : HTTP/2.0 206 Partial Content
Content-Length: 67108864
Accept-Ranges: bytes
Cf-Cache-Status: DYNAMIC
Cf-Ray: 9930364cea67fa35-BUD
Content-Range: bytes 402653184-469762047/572674039099
Content-Type: application/octet-stream
Date: Thu, 23 Oct 2025 09:29:48 GMT
Etag: "22ee711f68ab7319fbe42a9c08940e9a-10000"
Last-Modified: Sun, 19 Oct 2025 22:19:25 GMT
Nel: {"report_to":"cf-nel","success_fraction":0.0,"max_age":604800}
Report-To: {"group":"cf-nel","max_age":604800,"endpoints":[{"url":"https://a.nel.cloudflare.com/report/v4?s=7csvNm2TaSzgjj9oqD5xoAulDMARWGKp4nQRtD5t3z3iom0V6zR3aDkRkmHlpGIbfrB93LpsCT8iroByI9g3J4PCvW314vvOT83%2BZyxS9Fhq%2BRLk"}]}
Server: cloudflare
Vary: Accept-Encoding

2025/10/23 11:29:48 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/10/23 11:29:48 DEBUG : planet.pmtiles.cdabed0c.partial: writing chunk 6
2025/10/23 11:29:48 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/10/23 11:29:48 DEBUG : HTTP RESPONSE (req 0x1400067e280)
2025/10/23 11:29:48 DEBUG : HTTP/2.0 206 Partial Content
Content-Length: 67108864
Accept-Ranges: bytes
Cf-Cache-Status: DYNAMIC
Cf-Ray: 9930364cea6efa35-BUD
Content-Range: bytes 469762048-536870911/572674039099
Content-Type: application/octet-stream
Date: Thu, 23 Oct 2025 09:29:48 GMT
Etag: "22ee711f68ab7319fbe42a9c08940e9a-10000"
Last-Modified: Sun, 19 Oct 2025 22:19:25 GMT
Nel: {"report_to":"cf-nel","success_fraction":0.0,"max_age":604800}
Report-To: {"group":"cf-nel","max_age":604800,"endpoints":[{"url":"https://a.nel.cloudflare.com/report/v4?s=sclwWKHCX5QW9x1UkGbxJQiZy%2FD72n90m0y0sd6UelF%2F2ezu3TZMUkjBNcSU19ISeQvIzCwsQKsATZpM0WS9Z8J2UIEei7%2FGkbfqENSderAPzgKp"}]}
Server: cloudflare
Vary: Accept-Encoding

2025/10/23 11:29:48 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/10/23 11:29:48 DEBUG : planet.pmtiles.cdabed0c.partial: writing chunk 7
^C2025/10/23 11:29:52 INFO  : Signal received: interrupt
2025/10/23 11:29:52 INFO  : planet.pmtiles.cdabed0c.partial: Removing failed copy
2025/10/23 11:29:52 DEBUG : planet.pmtiles: multi-thread copy: cancelling transfer on exit
2025/10/23 11:29:52 DEBUG : planet.pmtiles: Reopening on read failure after offset 10485086 bytes: retry 1/10: context canceled
2025/10/23 11:29:52 DEBUG : planet.pmtiles: Reopening on read failure after offset 10766332 bytes: retry 1/10: context canceled
2025/10/23 11:29:52 DEBUG : planet.pmtiles: Reopening on read failure after offset 9676384 bytes: retry 1/10: context canceled
2025/10/23 11:29:52 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/10/23 11:29:52 DEBUG : HTTP REQUEST (req 0x14000830dc0)
2025/10/23 11:29:52 DEBUG :
2025/10/23 11:29:52 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/10/23 11:29:52 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/10/23 11:29:52 DEBUG : HTTP RESPONSE (req 0x14000830dc0)
2025/10/23 11:29:52 DEBUG : Error: context canceled
2025/10/23 11:29:52 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/10/23 11:29:52 DEBUG : planet.pmtiles: Reopen failed after offset 10485086 bytes read: Open failed: Get "https://download.mapterhorn.com/planet.pmtiles": context canceled
2025/10/23 11:29:52 DEBUG : planet.pmtiles: multi-thread copy: chunk 7/8534 failed: multi-thread copy: failed to write chunk: context canceled
2025/10/23 11:29:52 DEBUG : planet.pmtiles: Reopening on read failure after offset 11853824 bytes: retry 1/10: context canceled
2025/10/23 11:29:52 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/10/23 11:29:52 DEBUG : HTTP REQUEST (req 0x140004c0000)
2025/10/23 11:29:52 DEBUG :
2025/10/23 11:29:52 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/10/23 11:29:52 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/10/23 11:29:52 DEBUG : planet.pmtiles: multi-thread copy: chunk 9/8534 (536870912-603979776) size 64Mi starting
2025/10/23 11:29:52 DEBUG : HTTP RESPONSE (req 0x140004c0000)
2025/10/23 11:29:52 DEBUG : Error: context canceled
2025/10/23 11:29:52 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/10/23 11:29:52 DEBUG : planet.pmtiles: Reopen failed after offset 9676384 bytes read: Open failed: Get "https://download.mapterhorn.com/planet.pmtiles": context canceled
2025/10/23 11:29:52 DEBUG : planet.pmtiles: multi-thread copy: chunk 8/8534 failed: multi-thread copy: failed to write chunk: context canceled
2025/10/23 11:29:52 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/10/23 11:29:52 DEBUG : HTTP REQUEST (req 0x1400067edc0)
2025/10/23 11:29:52 DEBUG :
2025/10/23 11:29:52 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/10/23 11:29:52 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/10/23 11:29:52 DEBUG : HTTP RESPONSE (req 0x1400067edc0)
2025/10/23 11:29:52 DEBUG : Error: context canceled
2025/10/23 11:29:52 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/10/23 11:29:52 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/10/23 11:29:52 DEBUG : HTTP REQUEST (req 0x140003ba000)
2025/10/23 11:29:52 DEBUG :
2025/10/23 11:29:52 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/10/23 11:29:52 DEBUG : planet.pmtiles: Reopen failed after offset 10766332 bytes read: Open failed: Get "https://download.mapterhorn.com/planet.pmtiles": context canceled
2025/10/23 11:29:52 DEBUG : planet.pmtiles: multi-thread copy: chunk 6/8534 failed: multi-thread copy: failed to write chunk: context canceled
2025/10/23 11:29:52 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/10/23 11:29:52 DEBUG : HTTP RESPONSE (req 0x140003ba000)
2025/10/23 11:29:52 DEBUG : Error: context canceled
2025/10/23 11:29:52 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/10/23 11:29:52 DEBUG : planet.pmtiles: Reopen failed after offset 11853824 bytes read: Open failed: Get "https://download.mapterhorn.com/planet.pmtiles": context canceled
2025/10/23 11:29:52 DEBUG : planet.pmtiles: multi-thread copy: chunk 5/8534 failed: multi-thread copy: failed to write chunk: context canceled
2025/10/23 11:29:52 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/10/23 11:29:52 DEBUG : HTTP REQUEST (req 0x14000831040)
2025/10/23 11:29:52 DEBUG :
2025/10/23 11:29:52 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/10/23 11:29:52 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/10/23 11:29:52 DEBUG : HTTP RESPONSE (req 0x14000831040)
2025/10/23 11:29:52 DEBUG : Error: context canceled
2025/10/23 11:29:52 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/10/23 11:29:52 DEBUG : planet.pmtiles: multi-thread copy: chunk 9/8534 failed: multi-thread copy: failed to open source: Open failed: Get "https://download.mapterhorn.com/planet.pmtiles": context canceled
2025/10/23 11:29:52 DEBUG : planet.pmtiles: multi-thread copy: abort failed: multi-thread copy: failed to find temp file when aborting chunk writer: object not found
2025/10/23 11:29:52 INFO  : Exiting...
9

OK, I made it! I mean I didn't do anything, rclone works perfectly for bigger-than-disk-file-size scenario!

Copy from http to s3 works, AND --s3-chunk-size synchronizes both the downloading range requests and uploading multi-part chunks!

'rclone',
'copy',
'--http-url',
base_url,
f':http:{path}',
destination,
'--multi-thread-streams=8',
'--s3-chunk-size=100M',
10

It only does either downloading or uploading at a time, but it's perfectly fine for the use case.

CleanShot 2025-10-23 at 12.35.00@2x
CleanShot 2025-10-23 at 12.35.00@2x882×608 66 KB

11

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.