BackBlaze B2: Couldn't decode error response: EOF

Help and Support
1

What is the problem you are having with rclone?

I'm trying to sync to a BackBlaze B2 as I've done for years. And all of a sudden there's one directory (that almost never changes) that I can no longer sync (other directories in the same bucket work fine). This is probably an issue with BackBlaze, whom I'm going to contact. But I'm wondering why rclone gives back Couldn't decode error response: EOF. Why not relay the 404 directly back to me through the logs?

Here's what I get in the logs using --log-level DEBUG --dump responses:

2022/11/20 08:26:59 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2022/11/20 08:26:59 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/11/20 08:26:59 DEBUG : HTTP REQUEST (req 0x1400073a000)
2022/11/20 08:26:59 DEBUG : GET /b2api/v1/b2_authorize_account HTTP/1.1
Host: api.backblazeb2.com
User-Agent: rclone/v1.60.1
Authorization: XXXX
Accept-Encoding: gzip

2022/11/20 08:26:59 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/11/20 08:27:00 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/11/20 08:27:00 DEBUG : HTTP RESPONSE (req 0x1400073a000)
2022/11/20 08:27:00 DEBUG : HTTP/1.1 200
Content-Length: 774
Cache-Control: max-age=0, no-cache, no-store
Content-Type: application/json;charset=UTF-8
Date: Sun, 20 Nov 2022 16:27:00 GMT

{
  "absoluteMinimumPartSize": 5000000,
  "accountId": "__REDACTED__",
  "allowed": {
    "bucketId": "__REDACTED__",
    "bucketName": "__REDACTED__",
    "capabilities": [
      "listAllBucketNames",
      "listBuckets",
      "readBuckets",
      "listFiles",
      "readFiles",
      "shareFiles",
      "writeFiles",
      "deleteFiles",
      "readBucketEncryption",
      "writeBucketEncryption"
    ],
    "namePrefix": null
  },
  "apiUrl": "https://api001.backblazeb2.com",
  "authorizationToken": "__REDACTED__",
  "downloadUrl": "https://f001.backblazeb2.com",
  "minimumPartSize": 100000000,
  "recommendedPartSize": 100000000,
  "s3ApiUrl": "https://s3.us-west-001.backblazeb2.com"
}
2022/11/20 08:27:00 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/11/20 08:27:00 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/11/20 08:27:00 DEBUG : HTTP REQUEST (req 0x1400073ab00)
2022/11/20 08:27:00 DEBUG : HEAD /file/__REDACTED__/5d6iipgve0b3dphe1nldh77tbd HTTP/1.1
Host: f001.backblazeb2.com
User-Agent: rclone/v1.60.1
Authorization: XXXX

2022/11/20 08:27:00 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/11/20 08:27:01 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/11/20 08:27:01 DEBUG : HTTP RESPONSE (req 0x1400073ab00)
2022/11/20 08:27:01 DEBUG : HTTP/1.1 404
Content-Length: 94
Cache-Control: max-age=0, no-cache, no-store
Content-Type: application/json;charset=utf-8
Date: Sun, 20 Nov 2022 16:27:00 GMT

2022/11/20 08:27:01 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/11/20 08:27:01 DEBUG : Couldn't decode error response: EOF
2022/11/20 08:27:01 ERROR : Attempt 1/3 failed with 1 errors and: object not found
2022/11/20 08:27:01 ERROR : Attempt 2/3 failed with 1 errors and: object not found
2022/11/20 08:27:01 ERROR : Attempt 3/3 failed with 1 errors and: object not found

This seems like a fairly standard 404 response. Could this error be relayed in the logs without my having to --dump responses?

Run the command 'rclone version' and share the full output of the command.

rclone v1.60.1
- os/version: darwin 12.6.1 (64 bit)
- os/kernel: 21.6.0 (arm64)
- os/type: darwin
- os/arch: arm64
- go/version: go1.19.3
- go/linking: dynamic
- go/tags: cmount noselfupdate

Installed via MacPorts.

Which cloud storage system are you using? (eg Google Drive)

BackBlaze B2

The command you were trying to run (eg rclone copy /tmp remote:tmp)

time -h -a -o /Users/__REDACTED__/Library/Logs/__REDACTED__/bak.sync.__REDACTED__.log nice -n 15 /opt/local/bin/rclone sync --log-level DEBUG --dump responses /ext/ b2-__REDACTED__-backup-secret:prio3

The rclone config contents with secrets removed.

[b2-__REDACTED__-backup]
type = b2
account = __REDACTED__
key = __REDACTED__
hard_delete = true
encoding = Slash,Del,Ctl,InvalidUtf8,Dot

[b2-__REDACTED__-backup-secret]
type = crypt
remote = b2-__REDACTED__-backup:__REDACTED__-backup
password = __REDACTED__
password2 = __REDACTED__

A log from the command with the -vv flag

See above.

2

A good question.

I looked through the code and as far as I can make out it is supposed to be returning a nice friendly "object not found" error here.

Ah, and looking at your log, that is what it does

3

Oh yeah I did see that at first. But I'm not sure that's sufficient, as I had no idea what that meant.

With that error message, I'm left to ponder: What object? On the local or the remote side? Why is this object expected to exist?

An error message that said: HEAD /file/__REDACTED__/5d6iipgve0b3dphe1nldh77tbd received a 404 response would answer these questions. Because I can look up that directory on the BackBlaze web site and see that it in fact does exist, and so clearly this is BackBlaze problem.

4

Ok, it turns out that this isn't BackBlaze's fault.

I looked at the rclone changelog to see what changed since rclone 1-2 versions prior, and this is maybe what's going on: something changed with respect to symlinks in --copy-links not following symlinks on 1.60.0 - #5 by Animosity022

That's cause my source is a symlink and that was working fine for a year and all of a sudden it no longer is.

Anyway I need to dig deeper to figure out exactly what's going on.

5

That was a bug which was fixed in v1.60.1

There could be other bugs around the same new code which was introduced in v1.59.

You don't appear to be using any filters which is where the bug kicked in.

If you can find an operation which works in a previous version of rclone (eg v1.58) and fails in v1.60.1 then I can investigate further.

6

I think the problem is with the fix in 1.60.1 itself.

I tried v1.58.1, v1.59.2, and v1.60.0 and they all worked as always, without complaining when my /ext/ src is a symlink.

But when I use v1.60.1, then I get the 404 on the HEAD call to BackBlaze. But it does work when I dereference the /ext/ symlink and just invoke rclone with /Volumes/ExtDisk/ which is not a symlink.

I don't use include filters, but I do use exclude filters via the RCLONE_EXCLUDE_FROM envvar. I also have export RCLONE_LINKS=true, export RCLONE_DELETE_EXCLUDED=true, and export RCLONE_FAST_LIST=true. My remote has:

hard_delete = true
encoding = Slash,Del,Ctl,InvalidUtf8,Dot

while its "secret counterpart" has:

type = crypt
filename_encryption = standard
directory_name_encryption = true

So v1.60.1 behaves differently with my real files.

However, I spent a long time trying to re-create a simple scenario with dummy files, directories, and symlinks for others to reproduce. Without RCLONE_LINKS=true, I got it to work once, but now that I try again, I just can't get the error anymore—all versions above behave the same. Then I tried with RCLONE_LINKS=true, and now all versions behave the same by getting the 404 error. So I'm quite confused and I don't know what's going on. So I may have to give up on trying to figure out what has changed in rclone recently.

In any case, I do have a workaround which is to use readlink on the source directory.

7

I can replicate this with

rclone copy -vv --links ext b2:rclone-test1/

lrwxrwxrwx 1 ncw  ncw           5 Nov 24 20:10  ext -> b2src
drwxrwxr-x 2 ncw  ncw        4096 Nov 24 20:10  b2src

I get the same error with all the rclone versions I tried.

On further inspection though, I'm not getting the 404 from b2 so it isn't exactly the same. The filters may be causing that.

I think the problem is the --links flag. This means translate symlinks into link.rclonelink however the source directory is a link and I think that is the trouble as you don't really want to copy just a single file ext.rclonelink do you?

So I think your best solution here is indeed readlink.

1 Like
8

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.