B2: Error on application key limited to a prefix

axstet · February 8, 2021, 6:42pm

What is the problem you are having with rclone?

I am trying to use Rclone with a B2 application key that is limited to bucket test and prefix a/ which does not seem to work. Using a B2 application key limited to bucket test only, works fine. Is there known issue with B2 application keys limited to a prefix?

What is your rclone version (output from `rclone version`)

rclone v1.54.0

os/arch: linux/arm64
go version: go1.15.7

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Linux 5.10.5-v8+ aarch64 GNU/Linux

Which cloud storage system are you using? (eg Google Drive)

B2

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone -vv lsd b2:test/a/

The rclone config contents with secrets removed.

[b2]
type = b2
account = ***
key = ***

A log from the command with the `-vv` flag

2021/02/08 19:23:38 DEBUG : rclone: Version "v1.54.0" starting with parameters ["rclone" "-vv" "lsd" "b2:test/a/"]
2021/02/08 19:23:38 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2021/02/08 19:23:38 DEBUG : Creating backend with remote "b2:test/a/"
2021/02/08 19:23:39 DEBUG : Couldn't decode error response: EOF
2021/02/08 19:23:39 DEBUG : B2 bucket test: Unauthorized: Unknown 401  (401 unknown)
2021/02/08 19:23:39 DEBUG : pacer: low level retry 1/10 (error Unknown 401  (401 unknown))
2021/02/08 19:23:39 DEBUG : pacer: Rate limited, increasing sleep to 20ms
...
2021/02/08 19:23:42 DEBUG : Couldn't decode error response: EOF
2021/02/08 19:23:42 DEBUG : B2 bucket test: Unauthorized: Unknown 401  (401 unknown)
2021/02/08 19:23:43 DEBUG : pacer: Reducing sleep to 10ms
2021/02/08 19:23:43 DEBUG : pacer: low level retry 10/10 (error Unknown 401  (401 unknown))
2021/02/08 19:23:43 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2021/02/08 19:23:43 Failed to create file system for "b2:test/a/": failed to HEAD for download: Unknown 401  (401 unknown)

asdffdsa · February 8, 2021, 7:01pm

hello and welcome to the forum,

what is the output of
rclone lsd b2: -vv --recursive

axstet · February 8, 2021, 7:10pm

2021/02/08 20:07:54 DEBUG : rclone: Version "v1.54.0" starting with parameters ["rclone" "lsd" "-vv" "--recursive" "b2:"]
2021/02/08 20:07:54 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2021/02/08 20:07:54 DEBUG : Creating backend with remote "b2:"
          -1 2021-02-08 20:07:56        -1 test
2021/02/08 20:07:56 ERROR : test/a/config: Entry doesn't belong in directory "test" (contains subdir) - ignoring
2021/02/08 20:07:56 ERROR : test/a/data: Entry doesn't belong in directory "test" (contains subdir) - ignoring
2021/02/08 20:07:56 ERROR : test/a/index: Entry doesn't belong in directory "test" (contains subdir) - ignoring
2021/02/08 20:07:56 ERROR : test/a/keys: Entry doesn't belong in directory "test" (contains subdir) - ignoring
2021/02/08 20:07:56 ERROR : test/a/snapshots: Entry doesn't belong in directory "test" (contains subdir) - ignoring
2021/02/08 20:07:56 DEBUG : 6 go routines active

ncw · February 8, 2021, 10:01pm

Can you try this with --dump bodies and post the result?

axstet · February 9, 2021, 8:06am

2021/02/09 09:02:20 DEBUG : rclone: Version "v1.54.0" starting with parameters ["rclone" "lsd" "-vv" "--dump" "bodies" "--low-level-retries" "1" "b2:test/a/"]
2021/02/09 09:02:20 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2021/02/09 09:02:20 DEBUG : Creating backend with remote "b2:test/a/"
2021/02/09 09:02:20 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2021/02/09 09:02:20 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/02/09 09:02:20 DEBUG : HTTP REQUEST (req 0x40003ef900)
2021/02/09 09:02:20 DEBUG : GET /b2api/v1/b2_authorize_account HTTP/1.1
Host: api.backblazeb2.com
User-Agent: rclone/v1.54.0
Authorization: XXXX
Accept-Encoding: gzip

2021/02/09 09:02:20 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/02/09 09:02:21 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/02/09 09:02:21 DEBUG : HTTP RESPONSE (req 0x40003ef900)
2021/02/09 09:02:21 DEBUG : HTTP/1.1 200
Content-Length: 595
Cache-Control: max-age=0, no-cache, no-store
Content-Type: application/json;charset=UTF-8
Date: Tue, 09 Feb 2021 08:02:21 GMT

{
  "absoluteMinimumPartSize": 5000000,
  "accountId": "***",
  "allowed": {
    "bucketId": "***",
    "bucketName": "test",
    "capabilities": [
      "listAllBucketNames",
      "listBuckets",
      "listFiles",
      "readFiles",
      "writeFiles"
    ],
    "namePrefix": "a/"
  },
  "apiUrl": "https://api003.backblazeb2.com",
  "authorizationToken": "***",
  "downloadUrl": "https://f003.backblazeb2.com",
  "minimumPartSize": 100000000,
  "recommendedPartSize": 100000000
}
2021/02/09 09:02:21 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/02/09 09:02:21 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/02/09 09:02:21 DEBUG : HTTP REQUEST (req 0x40000ee100)
2021/02/09 09:02:21 DEBUG : HEAD /file/test/a HTTP/1.1
Host: f003.backblazeb2.com
User-Agent: rclone/v1.54.0
Authorization: XXXX

2021/02/09 09:02:21 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/02/09 09:02:22 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/02/09 09:02:22 DEBUG : HTTP RESPONSE (req 0x40000ee100)
2021/02/09 09:02:22 DEBUG : HTTP/1.1 401
Content-Length: 62
Cache-Control: max-age=0, no-cache, no-store
Content-Type: application/json;charset=utf-8
Date: Tue, 09 Feb 2021 08:02:21 GMT

2021/02/09 09:02:22 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/02/09 09:02:22 DEBUG : Couldn't decode error response: EOF
2021/02/09 09:02:22 DEBUG : B2 bucket test: Unauthorized: Unknown 401  (401 unknown)
2021/02/09 09:02:22 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/02/09 09:02:22 DEBUG : HTTP REQUEST (req 0x40000ee900)
2021/02/09 09:02:22 DEBUG : GET /b2api/v1/b2_authorize_account HTTP/1.1
Host: api.backblazeb2.com
User-Agent: rclone/v1.54.0
Authorization: XXXX
Accept-Encoding: gzip

2021/02/09 09:02:22 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/02/09 09:02:22 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/02/09 09:02:22 DEBUG : HTTP RESPONSE (req 0x40000ee900)
2021/02/09 09:02:22 DEBUG : HTTP/1.1 200
Content-Length: 595
Cache-Control: max-age=0, no-cache, no-store
Content-Type: application/json;charset=UTF-8
Date: Tue, 09 Feb 2021 08:02:21 GMT

{
  "absoluteMinimumPartSize": 5000000,
  "accountId": "***",
  "allowed": {
    "bucketId": "***",
    "bucketName": "test",
    "capabilities": [
      "listAllBucketNames",
      "listBuckets",
      "listFiles",
      "readFiles",
      "writeFiles"
    ],
    "namePrefix": "a/"
  },
  "apiUrl": "https://api003.backblazeb2.com",
  "authorizationToken": "***",
  "downloadUrl": "https://f003.backblazeb2.com",
  "minimumPartSize": 100000000,
  "recommendedPartSize": 100000000
}
2021/02/09 09:02:22 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/02/09 09:02:22 DEBUG : pacer: low level retry 1/1 (error Unknown 401  (401 unknown))
2021/02/09 09:02:22 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2021/02/09 09:02:22 Failed to create file system for "b2:test/a/": failed to HEAD for download: Unknown 401  (401 unknown)

ncw · February 9, 2021, 10:04am

Thanks!

I think that is a bug. Funnily enough I fixed an identical bug in the s3 backend yesterday...

Try this

v1.55.0-beta.5165.8cc1fe711.fix-b2-limited-path on branch fix-b2-limited-path (uploaded in 15-30 mins)

axstet · February 9, 2021, 6:34pm

Thank you very much for the fast fix, @ncw!
The provided beta version works fine for me.

ncw · February 10, 2021, 3:30pm

Thanks for testing!

I've merged this to master now which means it will be in the latest beta in 15-30 mins and released in v1.54.1

ncw · February 11, 2021, 3:14pm

Oops forgot to actually push that - it should be building now - ready in 30 mins!

system · February 14, 2021, 3:14pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.