S3 ACL issues on Backblaze with multi-part upload

Suspected Bug
1

This is my testing following my previous question

What is the problem you are having with rclone?

Using B2 via S3 for multipart uploads (testing) is giving ACL issues.

What is your rclone version (output from rclone version)

rclone v1.52.0
- os/arch: darwin/amd64
- go version: go1.14.3

Which OS you are using and how many bits (eg Windows 7, 64 bit)

macOS 10.14.6

Which cloud storage system are you using? (eg Google Drive)

S3 via Backblaze (as opposed to B2 API)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone --config rclone.cfg      \
    --s3-disable-checksum       \
    --s3-upload-cutoff 1M       \
    --s3-chunk-size 5M          \
    --s3-upload-concurrency 4   \
    -vv                         \
    copyto random.bin  s3A:jgwrclonetest/random.bin

And the same thing with s3B

Config:

[s3A]
type = s3
provider = other
env_auth = false
access_key_id = **REDACTED**
secret_access_key = *REDACTED**
endpoint = s3.us-west-000.backblazeb2.com
acl = bucket-owner-full-control

[s3B]
type = s3
provider = other
env_auth = false
access_key_id = *REDACTED**
secret_access_key = *REDACTED**
endpoint = s3.us-west-000.backblazeb2.com
acl = private

Also tested without --s3-disable-checksum though the whole point is I want to see it

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

With s3A:

2020/06/02 09:37:11 DEBUG : rclone: Version "v1.52.0" starting with parameters ["rclone" "--config" "rclone.cfg" "--s3-disable-checksum" "--s3-upload-cutoff" "1M" "--s3-chunk-size" "5M" "--s3-upload-concurrency" "4" "-vv" "copyto" "random.bin" "s3A:jgwrclonetest/random.bin"]
2020/06/02 09:37:11 DEBUG : Using config file from "**REDACTED**"
2020/06/02 09:37:11 DEBUG : fs cache: renaming cache item "random.bin" to be canonical "**REDACTED**"
2020/06/02 09:37:11 DEBUG : fs cache: renaming cache item "s3A:jgwrclonetest/" to be canonical "s3A:jgwrclonetest"
2020/06/02 09:37:11 DEBUG : random.bin: Modification times differ by -10m29.977932095s: 2020-06-02 09:31:32.624884541 -0600 MDT, 2020-06-02 09:21:02.646952446 -0600 MDT
2020/06/02 09:37:11 DEBUG : random.bin: MD5 = 90490a1d66ba164012dd3baa3328765c (Local file system at **REDACTED**)
2020/06/02 09:37:11 DEBUG : random.bin: MD5 = 0d3be77889dec53c705843951cc2501d (S3 bucket jgwrclonetest)
2020/06/02 09:37:11 DEBUG : random.bin: MD5 differ
2020/06/02 09:37:11 ERROR : random.bin: Failed to copy: InvalidArgument: Backblaze does not support the canned acl 'bucket-owner-full-control'
	status code: 400, request id: 2a207a39980321e3, host id: aNy5mojcdNyI5QTljZFphMmGwZOg5YjPU
2020/06/02 09:37:11 ERROR : Attempt 1/3 failed with 1 errors and: InvalidArgument: Backblaze does not support the canned acl 'bucket-owner-full-control'
	status code: 400, request id: 2a207a39980321e3, host id: aNy5mojcdNyI5QTljZFphMmGwZOg5YjPU
2020/06/02 09:37:11 DEBUG : random.bin: Modification times differ by -10m29.977932095s: 2020-06-02 09:31:32.624884541 -0600 MDT, 2020-06-02 09:21:02.646952446 -0600 MDT
2020/06/02 09:37:11 DEBUG : random.bin: MD5 = 90490a1d66ba164012dd3baa3328765c (Local file system at **REDACTED**)
2020/06/02 09:37:11 DEBUG : random.bin: MD5 = 0d3be77889dec53c705843951cc2501d (S3 bucket jgwrclonetest)
2020/06/02 09:37:11 DEBUG : random.bin: MD5 differ
2020/06/02 09:37:11 ERROR : random.bin: Failed to copy: InvalidArgument: Backblaze does not support the canned acl 'bucket-owner-full-control'
	status code: 400, request id: 73bcdc7f39839f10, host id: aN9FmIzciNx85OTnNZBBhSmEoZI85FjO0
2020/06/02 09:37:11 ERROR : Attempt 2/3 failed with 1 errors and: InvalidArgument: Backblaze does not support the canned acl 'bucket-owner-full-control'
	status code: 400, request id: 73bcdc7f39839f10, host id: aN9FmIzciNx85OTnNZBBhSmEoZI85FjO0
2020/06/02 09:37:12 DEBUG : random.bin: Modification times differ by -10m29.977932095s: 2020-06-02 09:31:32.624884541 -0600 MDT, 2020-06-02 09:21:02.646952446 -0600 MDT
2020/06/02 09:37:12 DEBUG : random.bin: MD5 = 90490a1d66ba164012dd3baa3328765c (Local file system at **REDACTED**)
2020/06/02 09:37:12 DEBUG : random.bin: MD5 = 0d3be77889dec53c705843951cc2501d (S3 bucket jgwrclonetest)
2020/06/02 09:37:12 DEBUG : random.bin: MD5 differ
2020/06/02 09:37:12 ERROR : random.bin: Failed to copy: InvalidArgument: Backblaze does not support the canned acl 'bucket-owner-full-control'
	status code: 400, request id: c70438f1d1b654ba, host id: aN8tmBjfQNzc5PTlDZCNhHWFWZPg57DNV
2020/06/02 09:37:12 ERROR : Attempt 3/3 failed with 1 errors and: InvalidArgument: Backblaze does not support the canned acl 'bucket-owner-full-control'
	status code: 400, request id: c70438f1d1b654ba, host id: aN8tmBjfQNzc5PTlDZCNhHWFWZPg57DNV
2020/06/02 09:37:12 INFO  :
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.1s

2020/06/02 09:37:12 DEBUG : 4 go routines active
2020/06/02 09:37:12 Failed to copyto: InvalidArgument: Backblaze does not support the canned acl 'bucket-owner-full-control'
	status code: 400, request id: c70438f1d1b654ba, host id: aN8tmBjfQNzc5PTlDZCNhHWFWZPg57DNV

And with s3B:

2020/06/02 09:39:23 DEBUG : rclone: Version "v1.52.0" starting with parameters ["rclone" "--config" "rclone.cfg" "--s3-disable-checksum" "--s3-upload-cutoff" "1M" "--s3-chunk-size" "5M" "--s3-upload-concurrency" "4" "-vv" "copyto" "random.bin" "s3B:jgwrclonetest/random.bin"]
2020/06/02 09:39:23 DEBUG : Using config file from "**REDACTED**/rclone.cfg"
2020/06/02 09:39:23 DEBUG : fs cache: renaming cache item "random.bin" to be canonical "**REDACTED**"
2020/06/02 09:39:23 DEBUG : fs cache: renaming cache item "s3B:jgwrclonetest/" to be canonical "s3B:jgwrclonetest"
2020/06/02 09:39:23 DEBUG : random.bin: Modification times differ by -10m29.977932095s: 2020-06-02 09:31:32.624884541 -0600 MDT, 2020-06-02 09:21:02.646952446 -0600 MDT
2020/06/02 09:39:23 DEBUG : random.bin: MD5 = 90490a1d66ba164012dd3baa3328765c (Local file system at **REDACTED**)
2020/06/02 09:39:23 DEBUG : random.bin: MD5 = 0d3be77889dec53c705843951cc2501d (S3 bucket jgwrclonetest)
2020/06/02 09:39:23 DEBUG : random.bin: MD5 differ
2020/06/02 09:39:23 ERROR : random.bin: Failed to copy: multipart upload failed to initialise: InvalidArgument: Backblaze does not support the 'x-amz-acl' header
	status code: 400, request id: 326227602be84b9c, host id: adYlu+msEbvNvA3cHbh0=
2020/06/02 09:39:23 ERROR : Attempt 1/3 failed with 1 errors and: multipart upload failed to initialise: InvalidArgument: Backblaze does not support the 'x-amz-acl' header
	status code: 400, request id: 326227602be84b9c, host id: adYlu+msEbvNvA3cHbh0=
2020/06/02 09:39:23 DEBUG : random.bin: Modification times differ by -10m29.977932095s: 2020-06-02 09:31:32.624884541 -0600 MDT, 2020-06-02 09:21:02.646952446 -0600 MDT
2020/06/02 09:39:24 DEBUG : random.bin: MD5 = 90490a1d66ba164012dd3baa3328765c (Local file system at **REDACTED**)
2020/06/02 09:39:24 DEBUG : random.bin: MD5 = 0d3be77889dec53c705843951cc2501d (S3 bucket jgwrclonetest)
2020/06/02 09:39:24 DEBUG : random.bin: MD5 differ
2020/06/02 09:39:24 ERROR : random.bin: Failed to copy: multipart upload failed to initialise: InvalidArgument: Backblaze does not support the 'x-amz-acl' header
	status code: 400, request id: f25a8c44fc4431bf, host id: adVhuDmuLbipv6neibno=
2020/06/02 09:39:24 ERROR : Attempt 2/3 failed with 1 errors and: multipart upload failed to initialise: InvalidArgument: Backblaze does not support the 'x-amz-acl' header
	status code: 400, request id: f25a8c44fc4431bf, host id: adVhuDmuLbipv6neibno=
2020/06/02 09:39:24 DEBUG : random.bin: Modification times differ by -10m29.977932095s: 2020-06-02 09:31:32.624884541 -0600 MDT, 2020-06-02 09:21:02.646952446 -0600 MDT
2020/06/02 09:39:24 DEBUG : random.bin: MD5 = 90490a1d66ba164012dd3baa3328765c (Local file system at **REDACTED**)
2020/06/02 09:39:24 DEBUG : random.bin: MD5 = 0d3be77889dec53c705843951cc2501d (S3 bucket jgwrclonetest)
2020/06/02 09:39:24 DEBUG : random.bin: MD5 differ
2020/06/02 09:39:24 ERROR : random.bin: Failed to copy: multipart upload failed to initialise: InvalidArgument: Backblaze does not support the 'x-amz-acl' header
	status code: 400, request id: a5cb2bb9d2dd4c8c, host id: adcpuamvgbrxvynfOblE=
2020/06/02 09:39:24 ERROR : Attempt 3/3 failed with 1 errors and: multipart upload failed to initialise: InvalidArgument: Backblaze does not support the 'x-amz-acl' header
	status code: 400, request id: a5cb2bb9d2dd4c8c, host id: adcpuamvgbrxvynfOblE=
2020/06/02 09:39:24 INFO  :
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.2s

2020/06/02 09:39:24 DEBUG : 4 go routines active
2020/06/02 09:39:24 Failed to copyto: multipart upload failed to initialise: InvalidArgument: Backblaze does not support the 'x-amz-acl' header
	status code: 400, request id: a5cb2bb9d2dd4c8c, host id: adcpuamvgbrxvynfOblE=

What else?

  • When I do a vanilla upload of random.bin it works fine!
  • I read in the B2 Docs about the ACLs. Hence the two tests.
    • I also tried removing the line from the config manually and no change

Is this a bug or user error?

2

When I tried the b2 s3 compatible API It passed all the integration tests which include testing multipart uploads. However when I try it now I can see it is broken in the way you describe.

So it looks like b2 must have changed something... It seems a little unfriendly to error out if you set X-Amz-Acl. Rclone tries quite hard to make sure you have a sensible ACL defaulting to "private".

The upload succeeds if I comment out the code which sets the ACL.

I could make a special ACL, say "unset" which would cause rclone to miss the ACL headers off the upload.

It might be worth asking backblaze support about this or looking on their forum as I think it is a regression.

3

Ok. I submitted a ticket to them and linked them to this page

1 Like
4

Do post an update when you hear from them!

5

Thank you for writing in. It looks like you are using ACLs which we do not have full support for:

I replied that I think they should do a little bit more digging and confirm that it's not a change on their side and informed them that this used to work just fine.

If I do not hear back soon, I will probably try to get the attention of some of the people on their subreddit.

1 Like
6

They are escalating to an engineer. We will see!

1 Like
7

Hey @jwink3101 did you get an update as we have run into this issue, I see based on @ncw reply there is no current way to avoid sending this header.

8

I have not heard anything. I figured I’d wait a few more days and ask. Or maybe even ask on reddit where a lot of the top staff hang out.

9

I still think this is a bug in b2 that wasn't there when I tested it, however I made a workaround here - you need to set the acl to "unset" and it should work.

I don't like this workaround and I'd rather not merge it - hoping b2 will fix the problem at their end!

https://beta.rclone.org/branch/v1.52.1-057-gbfae383c-fix-b2-acl-beta/ (uploaded in 15-30 mins)

10

I just got this from them:

There is a bug preventing rclone compatibility with the S3 Compatible API that is being resolved next week. You should be able to use it successfully on Friday of next week

2 Likes
11

Hooray! Thanks for contacting them @jwink3101

12

Thanks for chasing them :slight_smile:

13

Hey guys, just wondering if Backblaze have given any updates on this? I've also run into this issue...

14

I'm under the impression this is working now - what commands did you try with which rclone version?

15

They told me it was fixed. If not, you may want to open a ticket or post more thorough logs here and see if it’s in your side (or rclone’s)

16

Hi guys,

rclone version:
rclone v1.52.2-232-gff843516-beta

  • os/arch: linux/amd64
  • go version: go1.14.6

rclone command:
rclone copy :s3:bucket-name/folder1 :s3:bucket-name/folder2 -vv --stats 2s --stats-one-line --retries 1 --transfers 14 --fast-list --s3-access-key-id xyz --s3-secret-access-key xyz --s3-region eu-central-xyz --s3-endpoint https://s3.eu-central-xyz.backblazeb2.com --s3-provider Other

rclone output:
2020/07/29 16:59:39 DEBUG : Using config file from "/xyz/rclone.conf"
2020/07/29 16:59:42 DEBUG : S3 bucket bucket-name path folder2: Waiting for checks to finish
2020/07/29 16:59:42 DEBUG : S3 bucket bucket-name path folder2: Waiting for transfers to finish
2020/07/29 16:59:42 ERROR : test2/loading/test2.txt: Failed to copy: InvalidArgument: Backblaze does not support the 'x-amz-acl' header
status code: 400, request id: a56a63af04dbc399, host id: adUVunmvZbpVvvHdKbqM=
2020/07/29 16:59:43 INFO : 0 / 1.114 GBytes, 0%, 0 Bytes/s, ETA - (xfr#0/18)

2020/07/29 16:59:46 ERROR : destination/test3 (copy).zip: Failed to copy: InvalidArgument: Backblaze does not support the 'x-amz-acl' header
status code: 400, request id: 83eb1c1f64cb19d6, host id: adW1uu2vPbrZv+Hc0bvU=
2020/07/29 16:59:46 INFO : There was nothing to transfer
2020/07/29 16:59:46 ERROR : Attempt 1/1 failed with 19 errors and: InvalidArgument: Backblaze does not support the 'x-amz-acl' header
status code: 400, request id: 83eb1c1f64cb19d6, host id: adW1uu2vPbrZv+Hc0bvU=
2020/07/29 16:59:46 INFO : 0 / 0 Bytes, -, 0 Bytes/s, ETA -

2020/07/29 16:59:46 DEBUG : 4 go routines active
2020/07/29 16:59:46 Failed to copy with 19 errors: last error was: InvalidArgument: Backblaze does not support the 'x-amz-acl' header
status code: 400, request id: 83eb1c1f64cb19d6, host id: adW1uu2vPbrZv+Hc0bvU=

17

Ah, so you are doing a server side copy.... I think the previous report for this was for uploads.

You can demonstrate this quite easily with

rclone moveto -vv b2s3:rclone-test-bucket/10M b2s3:rclone-test-bucket/10M-2 --retries 1 --dump bodies

Which gives

2020/07/29 16:21:02 DEBUG : HTTP REQUEST (req 0xc0006e0100)
2020/07/29 16:21:02 DEBUG : PUT /rclone-test-bucket/10M-2 HTTP/1.1
Host: s3.us-west-001.backblazeb2.com
User-Agent: rclone/v1.52.2-DEV
Content-Length: 0
Authorization: XXXX
X-Amz-Acl: private
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Copy-Source: rclone-test-bucket/10M
X-Amz-Date: 20200729T152102Z
X-Amz-Metadata-Directive: COPY
Accept-Encoding: gzip


2020/07/29 16:21:02 DEBUG : HTTP RESPONSE (req 0xc0006e0100)
2020/07/29 16:21:02 DEBUG : HTTP/1.1 400 
Connection: close
Content-Length: 179
Cache-Control: max-age=0, no-cache, no-store
Content-Type: application/xml
Date: Wed, 29 Jul 2020 15:21:02 GMT
X-Amz-Id-2: adaNuIGs2biVvUXeIbkc=
X-Amz-Request-Id: 0370078cabcbc79e

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Error>
    <Code>InvalidArgument</Code>
    <Message>Backblaze does not support the 'x-amz-acl' header</Message>
</Error>

I think that would be worth pointing out to backblaze support since they appear to have put a workaround for rclone supplying ACLs when uploading.

18

Of course, sorry, I completely missed that the previous report was for uploads only...
The strange thing is that a couple of days ago, this was still working I believe (see posts in this other topic)

@jwink3101, is it possible to add this to your Backblaze bug report? I think they should fix it for all actions...

19

It was support ticket #575867.

I hope that helps!

I've also seen some wonky behavior with B2 and server-side-copy of large files but it's been a while and I don't recall if it was vanilla B2 or B2 via S3. I will eventually research it and make a github issue if I can isolate it

20

I love Backblaze but it is also a little bit frustrating that just yesterday, they quoted @ncw from twitter in their blog post saying it was "very compatible"

@ncw, out of curiosity, does it still pass with the same rating?