Backblaze B2 with Cloudflare worker conflicts on the authorization

Help and Support
1

TL;DR Adding workers to the cloudflare proxy, as in the official tutorial, seems to conflict with the authentication in rclone. The following is how I reached the speculation.

I was setting up rclone with backblaze b2, with RCLONE_B2_DOWNLOAD_URL set to Cloudflare to proxy requests. Meanwhile I followed the official blog from Backblaze "How to allow Cloudflare to fetch content from a Backblaze B2 private bucket – Backblaze Help"(couldn't post links) to enable public access to a private bucket.

I had a VPS working but on multiple clients rclone showed errors when copying an object.

2020/12/28 14:18:46 DEBUG : fs cache: adding new entry for parent of "<remote:file>", "<remote:path>"
2020/12/28 14:18:46 DEBUG : <file_name>: Need to transfer - File not found at Destination
2020/12/28 14:18:46 DEBUG : B2 bucket <bucket_name>: Unauthorized: Unknown 401 Unauthorized (401 bad_auth_token)
2020/12/28 14:18:46 DEBUG : pacer: low level retry 1/10 (error Unknown 401 Unauthorized (401 bad_auth_token))
2020/12/28 14:18:46 DEBUG : pacer: Rate limited, increasing sleep to 20ms

I did some search and found one post concerning Backblaze with Cloudflare. "Correct format for RCLONE_B2_DOWNLOAD_URL variable? - Help and Support - rclone forum" It wasn't the same case, but it came to me that it might due to the difference between versions. So I did a binary search on the releases. It happens that some commit between v1.52.3 and v1.53.0 breaks it. I looked up the commits between them and located a seemingly relevant commit 957311. I confirmed that with this commit reverted the problem resolves, but applying the revert to the latest commit does not do the same.

I added some debug info to confirm that this issues is raised by the different styles of api: fetch object by id or by name. I looked at the errors again and it came to me that maybe this has something to do with Cloudflare tampering the authentication headers. The worker script in the tutorial is as follows.

addEventListener('fetch', event => {
    event.respondWith(handleRequest(event.request))
})
async function handleRequest(request) {
let authToken='<b2 auth token>'
let b2Headers = new Headers(request.headers)
b2Headers.append("Authorization", authToken)
modRequest = new Request(request.url, {
    method: request.method,
    headers: b2Headers
})
const response = await fetch(modRequest)
return response
}

I removed the worker and everything got smooth. It took me a whole evening. Whew!

This post is for sharing my solution to people who encounter the same problems. I have some questions, too. The script enables public access to private buckets by settings the 'Authorization' header. I confirmed that with the worker I can publicly fetch the files from cloudflare. How did that conflict with the credentials in rclone? The files are requested by filenames but not file ids, so are there other authentication schemes?

I don't know golang, so apologies if I missed anything. Thanks in advance!

2

Interesting...

The problem seems to be that there are two Authentication headers? Is that right? One added by rclone and the other added by the workers script.

I have a feeling that the fact it works by ID but not by name might just be random luck as to which Auth header gets processed first.

I wonder if rclone should be adding auth at all when using a DownloadURL?

Yes that is what it looks like.

Good

I suspect that there were two Auth headers which confused b2 somehow.

Not as far as I know.

3

@ncw Thanks for your quick response & all the efforts on rclone!

Yes, it appears so. The endpoint for fetching file with ID was not loaded with the worker script and I suspect that's why it works.

Makes sense. I presumed that the header was overwritten, but a quick test with js api shows otherwise.

let myHeaders = new Headers();
myHeaders.append('Content-Type', 'image/jpeg');
myHeaders.get('Content-Type');
// "image/jpeg"
myHeaders.append('Content-Type', 'image/jpeg');
myHeaders.get('Content-Type');
// "image/jpeg, image/jpeg"

When the header exists, appending the header is not simply repeating the field. Instead, it adds a comma to separate the values, in which case there's no wonder that b2 doesn't recognize the authentication header.

It takes a line or two in the worker script to handle this case, so no big deal.

b2Headers.delete("Authorization")
// OR
if(!b2Headers.get("Authorization")){
    //...
}

I actually like the current behavior which keeps private buckets private. It just that I came across the b2 doc first and didn't know rclone uses auth for the DownloadURL. Thus the Cloudflare tweak to add auth header is not necessary. A small tip in the doc would be more than enough :slight_smile:

4

What would you suggest? Care to make an edit to the docs? You can do that via the web by clicking the edit button here https://github.com/rclone/rclone/blob/master/docs/content/b2.md (pencil at top right)

5

Sure. The specific part seems to be auto-generated so I modified backend/b2/b2.go instead. See github.com/rclone/rclone/compare/master...sshockwave:patch-1. I wonder if this slight change in doc is sufficient for a pr.

6

That looks perfect. Please send a PR :slight_smile:

7

Excuse me for the delay. I sent the PR!

1 Like
8

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.