S3 ACL issues on Backblaze with multi-part upload

Hey guys, just wondering if Backblaze have given any updates on this? I've also run into this issue...

I'm under the impression this is working now - what commands did you try with which rclone version?

They told me it was fixed. If not, you may want to open a ticket or post more thorough logs here and see if it’s in your side (or rclone’s)

Hi guys,

rclone version:
rclone v1.52.2-232-gff843516-beta

  • os/arch: linux/amd64
  • go version: go1.14.6

rclone command:
rclone copy :s3:bucket-name/folder1 :s3:bucket-name/folder2 -vv --stats 2s --stats-one-line --retries 1 --transfers 14 --fast-list --s3-access-key-id xyz --s3-secret-access-key xyz --s3-region eu-central-xyz --s3-endpoint https://s3.eu-central-xyz.backblazeb2.com --s3-provider Other

rclone output:
2020/07/29 16:59:39 DEBUG : Using config file from "/xyz/rclone.conf"
2020/07/29 16:59:42 DEBUG : S3 bucket bucket-name path folder2: Waiting for checks to finish
2020/07/29 16:59:42 DEBUG : S3 bucket bucket-name path folder2: Waiting for transfers to finish
2020/07/29 16:59:42 ERROR : test2/loading/test2.txt: Failed to copy: InvalidArgument: Backblaze does not support the 'x-amz-acl' header
status code: 400, request id: a56a63af04dbc399, host id: adUVunmvZbpVvvHdKbqM=
2020/07/29 16:59:43 INFO : 0 / 1.114 GBytes, 0%, 0 Bytes/s, ETA - (xfr#0/18)

2020/07/29 16:59:46 ERROR : destination/test3 (copy).zip: Failed to copy: InvalidArgument: Backblaze does not support the 'x-amz-acl' header
status code: 400, request id: 83eb1c1f64cb19d6, host id: adW1uu2vPbrZv+Hc0bvU=
2020/07/29 16:59:46 INFO : There was nothing to transfer
2020/07/29 16:59:46 ERROR : Attempt 1/1 failed with 19 errors and: InvalidArgument: Backblaze does not support the 'x-amz-acl' header
status code: 400, request id: 83eb1c1f64cb19d6, host id: adW1uu2vPbrZv+Hc0bvU=
2020/07/29 16:59:46 INFO : 0 / 0 Bytes, -, 0 Bytes/s, ETA -

2020/07/29 16:59:46 DEBUG : 4 go routines active
2020/07/29 16:59:46 Failed to copy with 19 errors: last error was: InvalidArgument: Backblaze does not support the 'x-amz-acl' header
status code: 400, request id: 83eb1c1f64cb19d6, host id: adW1uu2vPbrZv+Hc0bvU=

Ah, so you are doing a server side copy.... I think the previous report for this was for uploads.

You can demonstrate this quite easily with

rclone moveto -vv b2s3:rclone-test-bucket/10M b2s3:rclone-test-bucket/10M-2 --retries 1 --dump bodies

Which gives

2020/07/29 16:21:02 DEBUG : HTTP REQUEST (req 0xc0006e0100)
2020/07/29 16:21:02 DEBUG : PUT /rclone-test-bucket/10M-2 HTTP/1.1
Host: s3.us-west-001.backblazeb2.com
User-Agent: rclone/v1.52.2-DEV
Content-Length: 0
Authorization: XXXX
X-Amz-Acl: private
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Copy-Source: rclone-test-bucket/10M
X-Amz-Date: 20200729T152102Z
X-Amz-Metadata-Directive: COPY
Accept-Encoding: gzip

2020/07/29 16:21:02 DEBUG : HTTP RESPONSE (req 0xc0006e0100)
2020/07/29 16:21:02 DEBUG : HTTP/1.1 400 
Connection: close
Content-Length: 179
Cache-Control: max-age=0, no-cache, no-store
Content-Type: application/xml
Date: Wed, 29 Jul 2020 15:21:02 GMT
X-Amz-Id-2: adaNuIGs2biVvUXeIbkc=
X-Amz-Request-Id: 0370078cabcbc79e

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Error>
    <Code>InvalidArgument</Code>
    <Message>Backblaze does not support the 'x-amz-acl' header</Message>
</Error>

I think that would be worth pointing out to backblaze support since they appear to have put a workaround for rclone supplying ACLs when uploading.

Of course, sorry, I completely missed that the previous report was for uploads only...
The strange thing is that a couple of days ago, this was still working I believe (see posts in this other topic)

@jwink3101, is it possible to add this to your Backblaze bug report? I think they should fix it for all actions...

It was support ticket #575867.

I hope that helps!

I've also seen some wonky behavior with B2 and server-side-copy of large files but it's been a while and I don't recall if it was vanilla B2 or B2 via S3. I will eventually research it and make a github issue if I can isolate it

I love Backblaze but it is also a little bit frustrating that just yesterday, they quoted @ncw from twitter in their blog post saying it was "very compatible"

@ncw, out of curiosity, does it still pass with the same rating?

The ACL changes are after I did the test. I expect they'll fix it up quickly enough though!

Thanks guys, I will try to open a new ticket with them. Sorry, the link in my previous message was incorrect, it should have pointed to here

So what I am seeing is that running the server side copy command with a large file which requires multi-part copy works fine, but with a number of small files which don't require multi-part copy, it fails as described above...

I managed to replicate this.

2020/07/30 11:04:12 DEBUG : HTTP REQUEST (req 0xc00039d400)
2020/07/30 11:04:12 DEBUG : PUT /rclone-test-bucket/10M.copy8 HTTP/1.1
Host: s3.us-west-001.backblazeb2.com
User-Agent: rclone/v1.52.2-DEV
Content-Length: 0
Authorization: XXXX
X-Amz-Acl: private
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Copy-Source: rclone-test-bucket/10M
X-Amz-Date: 20200730T100412Z
X-Amz-Metadata-Directive: COPY
Accept-Encoding: gzip
2020/07/30 11:04:13 DEBUG : HTTP RESPONSE (req 0xc00039d400)
2020/07/30 11:04:13 DEBUG : HTTP/1.1 400 
Connection: close
Content-Length: 179
Cache-Control: max-age=0, no-cache, no-store
Content-Type: application/xml
Date: Thu, 30 Jul 2020 10:04:12 GMT
X-Amz-Id-2: addFu/Ws/bn5vgHeTboU=
X-Amz-Request-Id: 1576de9bf1ae1673

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Error>
    <Code>InvalidArgument</Code>
    <Message>Backblaze does not support the 'x-amz-acl' header</Message>
</Error>

Funnily enough X-Amz-Acl: private is perfectly acceptable on the multipart copy API so if you want a work-around you can use --s3-copy-cutoff 0. It isn't a great workaround as that API will do at minimum 2 API requests more but it is better than nothing.

Great, thanks for the confirmation! I will await a reply from Backblaze, hopefully they can sort this out quickly!

1 Like

Hi guys - just starting with Rclone and love it. Is there an update for the unsupported x-amz-acl error?

~$ rclone copy --s3-copy-cutoff 4G  b2pgs3: backups/pg/daily/7 b2pgs3: backups/pg/weekly/33
2020/08/24 06:49:10 ERROR : 3510.dat.gz: Failed to copy: InvalidArgument: Backblaze does not support the 'x-amz-acl' header
        status code: 400, request id: c1a4a8a99ca50cc7, host id: adfJuk2uCbj1vuHdfbhc=

Specifying --s3-copy-cutoff 0 gives a dive by zero error:

~$ rclone copy  --s3-copy-cutoff 0 b2pgs3:backups/pg/daily/7 b2pgs3:backups/pg/weekly/33
panic: runtime error: integer divide by zero

If I try the copy with the b2 backend, I get the ERROR : 3521.dat.gz: Failed to copy: Copy source too big: 5323717872 (400 bad_request)

~$ rclone copy b2pg:backups/pg/daily/7 b2pg:backups/pg/weekley/33
2020/08/24 06:52:19 ERROR : 3521.dat.gz: Failed to copy: Copy source too big: 5323717872 (400 bad_request)

I'm using the stable version:

~$ rclone --version
rclone v1.52.3
- os/arch: linux/amd64
- go version: go1.14.7

Thanks!

Hi @dkam!

So regarding the B2 S3 interface, server side copies still have that Backblaze does not support the 'x-amz-acl' header error. I logged a request with B2 a while ago already, this past Friday they handed the problem over to their engineers. I am awaiting further feedback from them...

Regarding the error you see using the normal B2 API, I assume that the code to fix that large file problem is not in the 1.52.3 release. You can try this beta v1.52.2-232-gff843516-beta maybe? This version is working on my side...

Thanks @ctonsing! I used v1.52.3-337-gd6996e33-beta and it's copied perfectly.

Great! :+1: :+1:

Just to report back: Backblaze said a while ago they'd look into the issue of not supporting the 'x-amz-acl' header for non-multi-part server-side copies. I haven't heard from them again, but according to my testing, this seems to be working again now. If anybody else could confirm, that would be great!

I can confirm it is working for me :slight_smile:

Thank you for confirming @ncw! :+1:t2: :+1:t2:

1 Like

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.