Backblaze b2 integration builds wrong URL for Cloudflare CDN

Help and Support
1

What is the problem you are having with rclone?

I'm a happy rclone user currently migrating from bare metal to backblaze b2. For this, I use the b2 integration. In front, I'm using Cloudflare as a CDN for caching and saving costs (cost saving only until October 3rd, that is).

My issue is that the B2 integration adds a /file/ as part of the path or the URI, which breaks the download_url pointing via Cloudflare to the CDN

Without the download URL it works fine.

This is the same for both the crypt wrapped and direct remote.

Run the command 'rclone version' and share the full output of the command.

On macOS:
rclone v1.65.0-beta.7392.b296f3780

  • os/version: darwin 13.5.2 (64 bit)
  • os/kernel: 22.6.0 (arm64)
  • os/type: darwin
  • os/arch: arm64 (ARMv8 compatible)
  • go/version: go1.21.1
  • go/linking: dynamic
  • go/tags: cmount

On my server:
rclone v1.64.0

  • os/version: alpine 3.18.3 (64 bit)
  • os/kernel: 6.1.0-0.deb11.11-amd64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.21.1
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Backblaze B2

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone -vv copyto --b2-download-url https://cdn.domain.tld  crypt-backblaze-<name>:/path/to/files /Users/<name>/test/

 rclone -vv  mount --b2-download-url https://cdn.domain.tld crypt-backblaze-<name>: /mnt/test

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

/data # rclone config redacted
[backblaze]
type = b2
account = XXX
key = XXX
hard_delete = true

[crypt-backblaze-<name>]
type = crypt
remote = backblaze:<name>
password = XXX
password2 = XXX

[crypt-backblaze-<othername>]
type = crypt
remote = backblaze:<othername>
password = XXX
password2 = XXX
### Double check the config for sensitive info before posting publicly

A log from the command that you were trying to run with the -vv flag

2023/09/28 15:23:10 DEBUG : DSC09652.jpg: Need to transfer - File not found at Destination
2023/09/28 15:23:10 DEBUG : DSC09658.jpg: Need to transfer - File not found at Destination
2023/09/28 15:23:10 DEBUG : DSC09666.jpg: Need to transfer - File not found at Destination
2023/09/28 15:23:10 DEBUG : DSC09704.jpg: Need to transfer - File not found at Destination
2023/09/28 15:23:10 DEBUG : DSC09751.jpg: Need to transfer - File not found at Destination
2023/09/28 15:23:10 DEBUG : DSC09762.jpg: Need to transfer - File not found at Destination
2023/09/28 15:23:10 DEBUG : DSC09770.jpg: Need to transfer - File not found at Destination
2023/09/28 15:23:10 DEBUG : DSC09774.jpg: Need to transfer - File not found at Destination
2023/09/28 15:23:10 DEBUG : DSC09781.jpg: Need to transfer - File not found at Destination
2023/09/28 15:23:10 DEBUG : Local file system at /Users/<name>/test: Waiting for checks to finish
2023/09/28 15:23:10 DEBUG : Local file system at /Users/<name>/test: Waiting for transfers to finish
2023/09/28 15:23:10 DEBUG : Couldn't decode error response: invalid character '<' looking for beginning of value
2023/09/28 15:23:10 ERROR : DSC09652.jpg: Failed to copy: failed to open source object: object not found
2023/09/28 15:23:10 DEBUG : Couldn't decode error response: invalid character '<' looking for beginning of value
2023/09/28 15:23:10 ERROR : DSC09658.jpg: Failed to copy: failed to open source object: object not found
2023/09/28 15:23:10 DEBUG : Couldn't decode error response: invalid character '<' looking for beginning of value
2023/09/28 15:23:10 ERROR : DSC09666.jpg: Failed to copy: failed to open source object: object not found
2023/09/28 15:23:10 DEBUG : Couldn't decode error response: invalid character '<' looking for beginning of value
2023/09/28 15:23:10 ERROR : DSC09704.jpg: Failed to copy: failed to open source object: object not found
2023/09/28 15:23:10 DEBUG : Couldn't decode error response: invalid character '<' looking for beginning of value
2023/09/28 15:23:10 ERROR : DSC09762.jpg: Failed to copy: failed to open source object: object not found
2023/09/28 15:23:10 DEBUG : Couldn't decode error response: invalid character '<' looking for beginning of value
2023/09/28 15:23:10 ERROR : DSC09770.jpg: Failed to copy: failed to open source object: object not found
2023/09/28 15:23:10 DEBUG : Couldn't decode error response: invalid character '<' looking for beginning of value
2023/09/28 15:23:10 ERROR : DSC09774.jpg: Failed to copy: failed to open source object: object not found

Checking the log from the worker, Links look like this:
https://cdn.domain.tld/file/<bucketname>/tur22lsuacq07gk5t2vij6ak9o/j05j1iqg95gbss0hkfuhod0hn4/9jk9iok21sukm6q57bteipu8nc

And following such a link yields:

<Error>
<Code>NoSuchBucket</Code>
<Message>The specified bucket does not exist: file</Message>
<Resource>file</Resource>
</Error>

If I remove /file from the path, it works. I get to download the correct, although encrypted file.

2

Hi @gymnae - Backblaze B2 has two APIs: its original API, that we now call 'B2 Native' and its S3-compatible API. As a result, each file can be accessed one of several ways, including:

  • By its 'friendly', pre-S3 URL, e.g.: https://f004.backblazeb2.com/file/metadaddy-public/hello.txt
  • By its S3 URL, virtual host-style: https://metadaddy-public.s3.us-west-004.backblazeb2.com/hello.txt
  • By its S3 URL, path-style: https://s3.us-west-004.backblazeb2.com/metadaddy-public/hello.txt

Rclone uses the the CDN domain that you specify with the path from the friendly URL, hence the /file in its request, but it looks like Cloudflare is proxying to the S3 API.

I think you can most easily solve this by just removing the /file prefix from the path in the worker. Assuming you've used this worker, in the file index.js, look for this line:

let path = url.pathname.replace(/^\//, '');

and replace it with:

let path = url.pathname.replace(/^\/file\//, '');

Where the worker was previously just removing the initial / from the path, now it will remove the initial /file/, and you'll end up with the path-style S3 URL that B2 is expecting.

If you're using a different worker, or you're otherwise stuck, just reply with the code that you see, and we can figure it out.

3

This has come up on the forum before.

I guess rclone could have a flag as to whether the --b2-download-url is a "B2 Native" URL or a "S3 Virtual Host" or "S3 Path Style" url?

I think the actually changes to the code would be quite small.

What do you think? Good idea? Either of you want to have a go at that?

4

Thanks for trying to help :slight_smile:
Yeah, I'm using the worker you linked and tried to alter the code with your new regex both per quick edit in the Browser and as a new deployment, but it didn't change the issue.
Here's a debug report from the worker:

{
  "outcome": "exception",
  "scriptName": "cloudflare-b2",
  "diagnosticsChannelEvents": [],
  "exceptions": [
    {
      "name": "AbortError",
      "message": "The operation was aborted",
      "timestamp": 1696103115950
    }
  ],
  "logs": [
    {
      "message": [
        "Range header in request for https://s3.eu-central-003.backblazeb2.com/file/<mybucket>/kmfjhljuajk9432c0e93a3st2c/0i35pl1fk6si128cavc6j2oip0/j6g8qlegah5i9q6recc9hprdug but no content-range header in response. Will retry 2 more times"
      ],
      "level": "error",
      "timestamp": 1696103115950
    }
  ],
  "eventTimestamp": 1696103115863,
  "event": {
    "request": {
      "url": "https://cdn.domain.tld/file/<mybucket>/kmfjhljuajk9432c0e93a3st2c/0i35pl1fk6si128cavc6j2oip0/j6g8qlegah5i9q6recc9hprdug",
      "method": "GET",
      "headers": {
        "accept-encoding": "gzip",
        "authorization": "REDACTED",
        "cf-connecting-ip": "89.58.28.37",
        "cf-ipcountry": "DE",
        "cf-ray": "80eef71a1e340f79",
        "cf-visitor": "{\"scheme\":\"https\"}",
        "connection": "Keep-Alive",
        "host": "cdn.domain.tld",
        "range": "bytes=0-247",
        "user-agent": "rclone/v1.64.0",
        "x-forwarded-proto": "https",
        "x-real-ip": "<removed>"
      },
      "cf": {
        "clientTcpRtt": 107,
        "longitude": "<removed>",
        "latitude": <removed>",
        "tlsCipher": "AEAD-AES128-GCM-SHA256",
        "continent": "EU",
        "asn": 197540,
        "country": "DE",
        "isEUCountry": "1",
        "tlsClientAuth": {
          "certIssuerDNLegacy": "",
          "certIssuerSKI": "",
          "certSubjectDNRFC2253": "",
          "certSubjectDNLegacy": "",
          "certFingerprintSHA256": "",
          "certNotBefore": "",
          "certSKI": "",
          "certSerial": "",
          "certIssuerDN": "",
          "certVerified": "NONE",
          "certNotAfter": "",
          "certSubjectDN": "",
          "certPresented": "0",
          "certRevoked": "0",
          "certIssuerSerial": "",
          "certIssuerDNRFC2253": "",
          "certFingerprintSHA1": ""
        },
        "tlsExportedAuthenticator": {
          "clientFinished": "af3547185107db9590f121410fe63aa68dd1e7746acba561cbcec56bde9ff41e",
          "clientHandshake": "e3ace95c12b985cdf3de4e308f082c161df02c42a889231f8efe416f508cd056",
          "serverHandshake": "6f5bf9ca53d4a0086ba20e8e73e00b81fc1f2a554d1f80345a89c7130c1d2de7",
          "serverFinished": "a684d6674c80414119cd5631313be843ce30943d050b6c85ed75321260bb9ad2"
        },
        "tlsVersion": "TLSv1.3",
        "colo": "EWR",
        "timezone": "Europe/Berlin",
        "edgeRequestKeepAliveStatus": 1,
        "requestPriority": "weight=16;exclusive=0;group=0;group-weight=0",
        "asOrganization": "netcup GmbH",
        "httpProtocol": "HTTP/2"
      }
    },
    "response": {
      "status": 500
    }
  },
  "id": 8
}

These are the URLs that backblaze shows to me:
Friendly URL: https://f003.backblazeb2.com/file/<my-bucket>
S3 URL: https://<my-bucket>.s3.eu-central003.backblazeb2.com
Native URL: https://f003.backblazeb2.com/b2api/v1/b2_download_file_by_id

But only .s3.eu-central003.backblazeb2.com is accepted by the worker

The S3 URL also doesn't include /file/, here's an example:
https://<my-bucket>.s3.eu-central-003.backblazeb2.com/2g3q1b4a05tu7vllki1tmd26pc/igdai20kr1739csngaggr3934c/2k8lt9p9j7h4ra4qsrhoqbc2j6b8rv8d9ukkoenk782pq0e0isqg

5

To reply to myself:
I might have misunderstood the description of the worker.
I thought it was to be used with the B2 type remote, but that didn't work.
Once I switched the remote to be of the type S3, it's working flawlessly.
I wondered why I would need to use the S3 endpoint, now I know. :slight_smile:

6

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.