Metadata not copied from B2 to Cloudflare R2

Help and Support
1

What is the problem you are having with rclone?

I need Content-Encoding:gzip to be preserved when sync from Backblaze B2 to Cloudflare R2 but --metadata flag doesn't work.

The command you were trying to run

rclone sync b2:b2test/1.txt r2:r2test/data --metadata --no-gzip-encoding

Run the command 'rclone version' and share the full output of the command.

rclone v1.59.2
- os/version: Microsoft Windows 10 Pro 21H2 (64 bit)
- os/kernel: 10.0.19044.1526 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.18.6
- go/linking: static
- go/tags: cmount

Which cloud storage system are you using?

Copy from Backblaze B2 to Cloudflare R2

A log from the command with the -vv flag

rclone sync b2:b2test/a/1.txt r2:r2test/data --metadata --no-gzip-encoding -vv
2022/10/04 20:48:25 DEBUG : rclone: Version "v1.59.2" starting with parameters ["rclone" "sync" "b2:b2test/a/1.txt" "r2:r2test/data" "--metadata" "--no-gzip-encoding" "-vv"]
2022/10/04 20:48:25 DEBUG : Creating backend with remote "b2:b2test/a/1.txt"
2022/10/04 20:48:25 DEBUG : Using config file from "C:\\Users\\BM\\AppData\\Roaming\\rclone\\rclone.conf"
2022/10/04 20:48:26 DEBUG : fs cache: adding new entry for parent of "b2:b2test/a/1.txt", "b2:b2test/a"
2022/10/04 20:48:26 DEBUG : Creating backend with remote "r2:r2test/data"
2022/10/04 20:48:27 DEBUG : 1.txt: Need to transfer - File not found at Destination
2022/10/04 20:48:30 INFO  : 1.txt: Copied (new)
2022/10/04 20:48:30 INFO  :
Transferred:        1.104 KiB / 1.104 KiB, 100%, 568 B/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:         4.8s

2022/10/04 20:48:30 DEBUG : 9 go routines active

rclone.conf

[b2]
type = b2
account = zzzzzz
key = zzzzzz

[r2]
type = s3
provider = Cloudflare
access_key_id = zzzzzz
secret_access_key = zzzzzz
endpoint = https://zzzzzz.r2.cloudflarestorage.com
acl = private
2

Note that the b2 backend doesn't have metadata support yet.

However you can access b2 via the s3 interface which does though, so I suggest you use that instead.

You shouldn't need this.

Note that R2 is a bit funny about Gzip encoding - it always decompresses objects when returning them I think. It causes one of the integration tests to fail.

1 Like
3

Thanks, I switched to s3 interface but it's still not working yet

[b2]
type = s3
provider = Other
access_key_id = zzz
secret_access_key = zzz
endpoint = s3.us-west-000.backblazeb2.com

There're content-encoding and cache-control in B2

image
image803×219 6.24 KB

But file synced to R2 doesn't have any of those.

image
image1329×535 13 KB

(I have to use --no-gzip-encoding because otherwise it failed with error net/http: HTTP/1.x transport connection broken: http: ContentLength=1131 with Body length 2144)

4

I tried this with the latest beta

First I created a file on s3 with some metadata

$ rclone lsjson --stat -M s3:rclone/gzip-encoded.txt
{
	"Path": "gzip-encoded.txt",
	"Name": "gzip-encoded.txt",
	"Size": 152,
	"MimeType": "text/plain; charset=utf-8",
	"ModTime": "2022-10-05T07:58:08.379747124+01:00",
	"IsDir": false,
	"Tier": "STANDARD",
	"Metadata": {
		"atime": "2022-10-05T08:00:16.399823648+01:00",
		"btime": "2022-10-05T07:02:27Z",
		"cache-control": "max-age=1234",
		"content-encoding": "gzip",
		"content-type": "text/plain; charset=utf-8",
		"gid": "1000",
		"mode": "100664",
		"mtime": "2022-10-05T07:58:08.379747124+01:00",
		"uid": "1000"
	}
}

I then copied it to b2

rclone copyto -M s3:rclone/gzip-encoded.txt b2s3:rclone-demo/gzip-encoded.txt

And it appears to have the right attributes on b2

$ rclone lsjson --stat -M b2s3:rclone-demo/gzip-encoded.txt
2022/10/05 08:07:25 NOTICE: s3: s3 provider "" not known - please set correctly
{
	"Path": "gzip-encoded.txt",
	"Name": "gzip-encoded.txt",
	"Size": 152,
	"MimeType": "text/plain;charset=utf-8",
	"ModTime": "2022-10-05T07:58:08.379747124+01:00",
	"IsDir": false,
	"Tier": "STANDARD",
	"Metadata": {
		"atime": "2022-10-05T08:00:16.399823648+01:00",
		"btime": "2022-10-05T07:06:19Z",
		"cache-control": "max-age=1234",
		"content-encoding": "gzip",
		"content-type": "text/plain;charset=utf-8",
		"gid": "1000",
		"mode": "100664",
		"mtime": "2022-10-05T07:58:08.379747124+01:00",
		"uid": "1000"
	}
}

You will need the latest beta to fix this.

1 Like
5

It worked, thank you a lot for very quick responses and fix!
(actually --metadata has always been copied correctly, it just that Cloudflare R2 UI does not display them)

1 Like
6

I think this behavior related to Transparent Decompression when the request doesn't have accept-encoding: gzip:

Vitali — 05/03/2022

  1. Just a heads up that we're going to be rolling out transparent decompression this week to the S3 & APIv4 endpoints.
    What does this mean? Transcoding of gzip-compressed files  |  Cloud Storage  |  Google Cloud
    Basically, if you have content-encoding: gzip and your request doesn't have accept-encoding: gzip, content gets decoded automatically unless cache-control has a value of no-transform. cache-control/content-encoding need not be set at upload time either - you can override cache-control (& any other predefined system metadata) when issuing the get request as per the S3 GetObject spec. (edited)
    This may break any of you that are using clients that don't send accept-encoding but are expecting the response back. We're eager for feedback if this breaks anyone.

If you store an object with Content-Encoding: gzip , there's three paths that transparent decompression can take:

  1. You have Accept-Encoding: gzip in your request headers, so the file is served as-is.
  2. You don't have an Accept-Encoding header (or possibly a value other than gzip ?), so it is decompressed on-the-fly.
  3. The object is stored with Cache-Control: no-transform , so the file is served as-is.

Link to the original messages is in Cloudflare discord: Discord , Discord

7

That is a useful work-around - thank you .

8

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.