Rclone Missing Attributes after copy and sync on Rados S3 buckets

morphin · March 6, 2021, 10:36am

What is the problem you are having with rclone?

I'm trying to move my 450M object data from Ceph Cluster Rados S3 to a new cluster S3.
I have some important attributes in the object headers and I tried to sync and copy these objects with following parameters below:

After the copy I notice that attrributes are not copied to destination.
I tried to compile 111 metadata branch and nothing changed.
After that I tried the rclone in my test zone with newly created testfile and vola! Attributes were ok and that makes me wonder what is the problem? Rclone can copy files with attributes in my test env with test objects (not the same objects) but when I try in my Prod env rclone can show the attributes in --dump-headers -vv but not moving to destination! Thats weird! "The only change was in test env I write attributes via rados setxattr. In prod env http writes that while writing an object."

These attributes checked with "radosgw-admin --id radosgw.$id object stat --bucket=$bucketname --object=$objectname"

Source attributes:
"attrs": {
"user.rgw.content_type": "application/octet-stream",
"user.rgw.pg_ver": "��\u0002",
"user.rgw.source_zone": "]�G",
"user.rgw.tail_tag": "c106b26b-5150-4fd6-9504-dee3ca5c0968.121436688.45753903",
"user.rgw.x-amz-content-sha256": "STREAMING-AWS4-HMAC-SHA256-PAYLOAD",
"user.rgw.x-amz-date": "20200616T101659Z",
"user.rgw.x-amz-decoded-content-length": "86272",
"user.rgw.x-amz-meta-feature-count": "674"
}

Destination attributes after copy or sync:
"attrs": {
"user.rgw.content_type": "application/octet-stream",
"user.rgw.pg_ver": "\u000c�\t",
"user.rgw.source_zone": "́\u0013�",
"user.rgw.tail_tag": "fe8ee939-f5db-4955-8d2b-12911b27a8c6.177085028.7727128",
"user.rgw.x-amz-acl": "private",
"user.rgw.x-amz-meta-mtime": "1592302619"
}

Also --dump headers -vv log is below.

I couldn't find a way and I don't know where should I look? I'm trying to copy 450M object and I have to deal with rados getxattr & setxattr every object after the copy. I've done that for 30M but thats a lot! And thats idiotic way to do! Can you help me please?

What is your rclone version (output from `rclone version`)

rclone v1.54.0

go version: go1.15.7

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Arch Linux 64bit

Which cloud storage system are you using? (eg Google Drive)

Ceph Cluster - Rados Gateway

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

`rclone copy --files-from "missing.files" prod:$bucketname new:$bucketname --log-file $bucketname-run.log -vv --progress --ignore-existing --fast-list --checksum --transfers 100 --checkers 100 --s3-list-chunk 2000 --no-traverse --s3-no-check-bucket --no-update-modtime`

`rclone sync prod:$bucketname new:$bucketname --log-file $bucketname-run.log --fast-list --checksum --s3-list-chunk 2000 -vv --progress --transfers 100 --checkers 100 --no-update-modtime`

The rclone config contents with secrets removed.

A log from the command with the `-vv` flag

rclone copy prod:$bucketname/2020/file.gz new:$bucketname/2020 -vv --no-update-modtime --s3-no-check-bucket --retries 1 --dump headers

2021/03/06 13:24:39 DEBUG : rclone: Version "v1.54.0" starting with parameters ["rclone" "copy" "prodabc:abc-v2/descriptor/2018/08/28/b20175f4-ca2c-47e5-a2b5-24acc16c2b39" "yeniabc:abc-v2/descriptor/2018/08/28/" "-vv" "--no-update-modtime" "--retries" "1" "--dump" "headers" "--no-check-dest" "--s3-no-check-bucket"]
2021/03/06 13:24:39 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2021/03/06 13:24:39 DEBUG : Creating backend with remote "prodabc:abc-v2/descriptor/2018/08/28/b20175f4-ca2c-47e5-a2b5-24acc16c2b39"
2021/03/06 13:24:39 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2021/03/06 13:24:39 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2021/03/06 13:24:39 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/06 13:24:39 DEBUG : HTTP REQUEST (req 0xc000307000)
2021/03/06 13:24:39 DEBUG : HEAD /abc-v2/descriptor/2018/08/28/b20175f4-ca2c-47e5-a2b5-24acc16c2b39 HTTP/1.1
Host: 10.151.106.101:8091
User-Agent: rclone/v1.54.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210306T102439Z

2021/03/06 13:24:39 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/06 13:24:39 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/06 13:24:39 DEBUG : HTTP RESPONSE (req 0xc000307000)
2021/03/06 13:24:39 DEBUG : HTTP/1.1 200 OK
Content-Length: 86272
Accept-Ranges: bytes
Connection: Keep-Alive
Content-Type: application/octet-stream
Date: Sat, 06 Mar 2021 10:24:39 GMT
Etag: "3000ab75d49f8c0d64cd1b52fbbec5ef"
Last-Modified: Tue, 16 Jun 2020 10:16:59 GMT
X-Amz-Meta-Feature-Count: 674
X-Amz-Request-Id: tx0000000000000000000e2-0060435867-1afb26ba-ank
X-Rgw-Object-Type: Normal

2021/03/06 13:24:39 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/06 13:24:39 DEBUG : fs cache: adding new entry for parent of "prodabc:abc-v2/descriptor/2018/08/28/b20175f4-ca2c-47e5-a2b5-24acc16c2b39", "prodabc:abc-v2/descriptor/2018/08/28"
2021/03/06 13:24:39 DEBUG : Creating backend with remote "yeniabc:abc-v2/descriptor/2018/08/28/"
2021/03/06 13:24:39 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2021/03/06 13:24:39 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2021/03/06 13:24:39 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/06 13:24:39 DEBUG : HTTP REQUEST (req 0xc000a90c00)
2021/03/06 13:24:39 DEBUG : HEAD /abc-v2/descriptor/2018/08/28 HTTP/1.1
Host: 10.151.106.172:8091
User-Agent: rclone/v1.54.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210306T102439Z

2021/03/06 13:24:39 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/06 13:24:39 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/06 13:24:39 DEBUG : HTTP RESPONSE (req 0xc000a90c00)
2021/03/06 13:24:39 DEBUG : HTTP/1.1 404 Not Found
Content-Length: 208
Accept-Ranges: bytes
Connection: Keep-Alive
Content-Type: application/xml
Date: Sat, 06 Mar 2021 10:24:39 GMT
X-Amz-Request-Id: tx000000000000000b797b9-0060435867-a8e1a64-prod

2021/03/06 13:24:39 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/06 13:24:39 DEBUG : fs cache: renaming cache item "yeniabc:abc-v2/descriptor/2018/08/28/" to be canonical "yeniabc:abc-v2/descriptor/2018/08/28"
2021/03/06 13:24:39 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/06 13:24:39 DEBUG : HTTP REQUEST (req 0xc000d24300)
2021/03/06 13:24:39 DEBUG : HEAD /abc-v2/descriptor/2018/08/28/b20175f4-ca2c-47e5-a2b5-24acc16c2b39 HTTP/1.1
Host: 10.151.106.101:8091
User-Agent: rclone/v1.54.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210306T102439Z

2021/03/06 13:24:39 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/06 13:24:39 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/06 13:24:39 DEBUG : HTTP RESPONSE (req 0xc000d24300)
2021/03/06 13:24:39 DEBUG : HTTP/1.1 200 OK
Content-Length: 86272
Accept-Ranges: bytes
Connection: Keep-Alive
Content-Type: application/octet-stream
Date: Sat, 06 Mar 2021 10:24:39 GMT
Etag: "3000ab75d49f8c0d64cd1b52fbbec5ef"
Last-Modified: Tue, 16 Jun 2020 10:16:59 GMT
X-Amz-Meta-Feature-Count: 674
X-Amz-Request-Id: tx0000000000000000000e3-0060435867-1afb26ba-ank
X-Rgw-Object-Type: Normal

2021/03/06 13:24:39 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/06 13:24:39 DEBUG : b20175f4-ca2c-47e5-a2b5-24acc16c2b39: Need to transfer - File not found at Destination
2021/03/06 13:24:39 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/06 13:24:39 DEBUG : HTTP REQUEST (req 0xc000307500)
2021/03/06 13:24:39 DEBUG : GET /abc-v2/descriptor/2018/08/28/b20175f4-ca2c-47e5-a2b5-24acc16c2b39 HTTP/1.1
Host: 10.151.106.101:8091
User-Agent: rclone/v1.54.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210306T102439Z
Accept-Encoding: gzip

2021/03/06 13:24:39 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/06 13:24:39 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/06 13:24:39 DEBUG : HTTP RESPONSE (req 0xc000307500)
2021/03/06 13:24:39 DEBUG : HTTP/1.1 200 OK
Content-Length: 86272
Accept-Ranges: bytes
Connection: Keep-Alive
Content-Type: application/octet-stream
Date: Sat, 06 Mar 2021 10:24:39 GMT
Etag: "3000ab75d49f8c0d64cd1b52fbbec5ef"
Last-Modified: Tue, 16 Jun 2020 10:16:59 GMT
**X-Amz-Meta-Feature-Count: 1674**
X-Amz-Request-Id: tx0000000000000000000e4-0060435867-1afb26ba-ank
X-Rgw-Object-Type: Normal

2021/03/06 13:24:39 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/06 13:24:39 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/06 13:24:39 DEBUG : HTTP REQUEST (req 0xc000a7d300)
2021/03/06 13:24:39 DEBUG : PUT /abc-v2/descriptor/2018/08/28/b20175f4-ca2c-47e5-a2b5-24acc16c2b39?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=A6T7AOXGXC9YD667UBJ2%2F20210306%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210306T102439Z&X-Amz-Expires=900&X-Amz-SignedHeaders=content-md5%3Bcontent-type%3Bhost%3Bx-amz-acl%3Bx-amz-meta-mtime&X-Amz-Signature=bad67d5c96a0f024d11faaf4a742bc735d05510208ab0a73b818d6b53a202e70 HTTP/1.1
Host: 10.151.106.172:8091
User-Agent: rclone/v1.54.0
Content-Length: 86272
content-md5: MACrddSfjA1kzRtS+77F7w==
content-type: application/octet-stream
x-amz-acl: private
x-amz-meta-mtime: 1592302619
Accept-Encoding: gzip

2021/03/06 13:24:39 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/06 13:24:39 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/06 13:24:39 DEBUG : HTTP RESPONSE (req 0xc000a7d300)
2021/03/06 13:24:39 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Accept-Ranges: bytes
Connection: Keep-Alive
Date: Sat, 06 Mar 2021 10:24:39 GMT
Etag: "3000ab75d49f8c0d64cd1b52fbbec5ef"
X-Amz-Request-Id: tx000000000000000b797ba-0060435867-a8e1a64-prod
X-Amz-Version-Id: 68-maogHYV0OFIrSyhSUv4YpIiUdH7b

2021/03/06 13:24:39 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/06 13:24:39 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/06 13:24:39 DEBUG : HTTP REQUEST (req 0xc000a91400)
2021/03/06 13:24:39 DEBUG : HEAD /abc-v2/descriptor/2018/08/28/b20175f4-ca2c-47e5-a2b5-24acc16c2b39 HTTP/1.1
Host: 10.151.106.172:8091
User-Agent: rclone/v1.54.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210306T102439Z

2021/03/06 13:24:39 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/06 13:24:39 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/06 13:24:39 DEBUG : HTTP RESPONSE (req 0xc000a91400)
2021/03/06 13:24:39 DEBUG : HTTP/1.1 200 OK
Content-Length: 86272
Accept-Ranges: bytes
Connection: Keep-Alive
Content-Type: application/octet-stream
Date: Sat, 06 Mar 2021 10:24:39 GMT
Etag: "3000ab75d49f8c0d64cd1b52fbbec5ef"
Last-Modified: Sat, 06 Mar 2021 10:24:39 GMT
X-Amz-Meta-Mtime: 1592302619
X-Amz-Request-Id: tx000000000000000b797bb-0060435867-a8e1a64-prod
X-Amz-Version-Id: 68-maogHYV0OFIrSyhSUv4YpIiUdH7b
X-Rgw-Object-Type: Normal

2021/03/06 13:24:39 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/06 13:24:39 DEBUG : b20175f4-ca2c-47e5-a2b5-24acc16c2b39: MD5 = 3000ab75d49f8c0d64cd1b52fbbec5ef OK
2021/03/06 13:24:39 INFO  : b20175f4-ca2c-47e5-a2b5-24acc16c2b39: Copied (new)
2021/03/06 13:24:39 INFO  :
Transferred:   	   84.250k / 84.250 kBytes, 100%, 5.852 MBytes/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:         0.0s

2021/03/06 13:24:39 DEBUG : 8 go routines active

ncw · March 6, 2021, 11:34am

With the fix-111-metadata branch, rclone will copy metadata that looks like this.

Is this metadata set by CEPH? I don't know how to read that. If it doesn't appear in the HEAD request then rclone won't copy it.

morphin · March 7, 2021, 7:51am

Hello ncw! First of all; I tried everything to move my massive storage. And I can say that Rclone is the best and fastest solution so far! I moved 30M object under 15 hour I admire your work! You're awesome!

No the metadata written by using s3. This is the code: objMetaData.setUserMetadata(new HashMap<String, String>(){{put("feature-count", String.valueOf(featureNum));}});

In the attached debug log I can see the metadata in request with --dump headers:

Etag: "3000ab75d49f8c0d64cd1b52fbbec5ef"
Last-Modified: Tue, 16 Jun 2020 10:16:59 GMT
X-Amz-Meta-Feature-Count: 674
X-Amz-Request-Id: tx0000000000000000000e2-0060435867-1afb26ba-ank
X-Rgw-Object-Type: Normal

I think Latest Rclone able to read but can't or won't write the metadata to destination for some reason.

I think I was very tired last night. I build the fix 111-metadata branch but failed to try it somehow
Today I gave an another chance and its working if I delete or re-write the object at destination.

Source metadata:

"attrs": {
    "user.rgw.content_type": "application/octet-stream",
    "user.rgw.pg_ver": "��\u0002",
    "user.rgw.source_zone": "]�G",
    "user.rgw.tail_tag": "c106b26b-5150-4fd6-9504-dee3ca5c0968.121436688.45753903",
    "user.rgw.x-amz-content-sha256": "STREAMING-AWS4-HMAC-SHA256-PAYLOAD",
    "user.rgw.x-amz-date": "20200616T101659Z",
    "user.rgw.x-amz-decoded-content-length": "86272",
    "user.rgw.x-amz-meta-feature-count": "674"

Destination metadata:

"attrs": {
    "user.rgw.content_type": "application/octet-stream",
    "user.rgw.pg_ver": "T\u001d\n",
    "user.rgw.source_zone": "́\u0013�",
    "user.rgw.tail_tag": "fe8ee939-f5db-4955-8d2b-12911b27a8c6.177085028.12882738",
    "user.rgw.x-amz-acl": "private",
    "user.rgw.x-amz-meta-feature-count": "674",
    "user.rgw.x-amz-meta-mtime": "1592302619"
}

Rclone "fix-111-metadata branch" DEBUG Log:

2021/03/07 11:32:15 DEBUG : rclone: Version "v1.54.0-DEV" starting with parameters ["./rclone" "sync" "prod:abc/descriptor/2018/08/28/b20175f4-ca2c-47e5-a2b5-24acc16c2b39" "new:abc/descriptor/2018/08/28/" "-vv" "--no-update-modtime" "--retries" "1" "--dump" "headers" "--s3-no-check-bucket"]
2021/03/07 11:32:15 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2021/03/07 11:32:15 DEBUG : Creating backend with remote "prod:abc/descriptor/2018/08/28/b20175f4-ca2c-47e5-a2b5-24acc16c2b39"
2021/03/07 11:32:15 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2021/03/07 11:32:15 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/07 11:32:15 DEBUG : HTTP REQUEST (req 0xc00052d100)
2021/03/07 11:32:15 DEBUG : HEAD /abc/descriptor/2018/08/28/b20175f4-ca2c-47e5-a2b5-24acc16c2b39 HTTP/1.1
Host: 10.151.106.101:8091
User-Agent: rclone/v1.54.0-DEV
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210307T083215Z

2021/03/07 11:32:15 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/07 11:32:15 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/07 11:32:15 DEBUG : HTTP RESPONSE (req 0xc00052d100)
2021/03/07 11:32:15 DEBUG : HTTP/1.1 200 OK
Content-Length: 86272
Accept-Ranges: bytes
Connection: Keep-Alive
Content-Type: application/octet-stream
Date: Sun, 07 Mar 2021 08:32:15 GMT
Etag: "3000ab75d49f8c0d64cd1b52fbbec5ef"
Last-Modified: Tue, 16 Jun 2020 10:16:59 GMT
X-Amz-Meta-Feature-Count: 674
X-Amz-Request-Id: tx0000000000000000000fe-0060448f8f-1afb26ba-ank
X-Rgw-Object-Type: Normal

2021/03/07 11:32:15 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/07 11:32:15 DEBUG : fs cache: adding new entry for parent of "prod:abc/descriptor/2018/08/28/b20175f4-ca2c-47e5-a2b5-24acc16c2b39", "prod:abc/descriptor/2018/08/28"
2021/03/07 11:32:15 DEBUG : Creating backend with remote "new:abc/descriptor/2018/08/28/"
2021/03/07 11:32:15 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/07 11:32:15 DEBUG : HTTP REQUEST (req 0xc00052da00)
2021/03/07 11:32:15 DEBUG : HEAD /abc/descriptor/2018/08/28 HTTP/1.1
Host: 10.151.106.172:8091
User-Agent: rclone/v1.54.0-DEV
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210307T083215Z

2021/03/07 11:32:15 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/07 11:32:15 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/07 11:32:15 DEBUG : HTTP RESPONSE (req 0xc00052da00)
2021/03/07 11:32:15 DEBUG : HTTP/1.1 404 Not Found
Content-Length: 208
Accept-Ranges: bytes
Connection: Keep-Alive
Content-Type: application/xml
Date: Sun, 07 Mar 2021 08:32:15 GMT
X-Amz-Request-Id: tx000000000000000c49330-0060448f8f-a8e1a64-prod

2021/03/07 11:32:15 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/07 11:32:15 DEBUG : fs cache: renaming cache item "new:abc/descriptor/2018/08/28/" to be canonical "new:abc/descriptor/2018/08/28"
2021/03/07 11:32:15 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/07 11:32:15 DEBUG : HTTP REQUEST (req 0xc0007d0200)
2021/03/07 11:32:15 DEBUG : HEAD /abc/descriptor/2018/08/28/b20175f4-ca2c-47e5-a2b5-24acc16c2b39 HTTP/1.1
Host: 10.151.106.101:8091
User-Agent: rclone/v1.54.0-DEV
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210307T083215Z

2021/03/07 11:32:15 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/07 11:32:15 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/07 11:32:15 DEBUG : HTTP RESPONSE (req 0xc0007d0200)
2021/03/07 11:32:15 DEBUG : HTTP/1.1 200 OK
Content-Length: 86272
Accept-Ranges: bytes
Connection: Keep-Alive
Content-Type: application/octet-stream
Date: Sun, 07 Mar 2021 08:32:15 GMT
Etag: "3000ab75d49f8c0d64cd1b52fbbec5ef"
Last-Modified: Tue, 16 Jun 2020 10:16:59 GMT
X-Amz-Meta-Feature-Count: 674
X-Amz-Request-Id: tx0000000000000000000ff-0060448f8f-1afb26ba-ank
X-Rgw-Object-Type: Normal

2021/03/07 11:32:15 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/07 11:32:15 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/07 11:32:15 DEBUG : HTTP REQUEST (req 0xc0007d0500)
2021/03/07 11:32:15 DEBUG : HEAD /abc/descriptor/2018/08/28/b20175f4-ca2c-47e5-a2b5-24acc16c2b39 HTTP/1.1
Host: 10.151.106.172:8091
User-Agent: rclone/v1.54.0-DEV
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210307T083215Z

2021/03/07 11:32:15 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/07 11:32:15 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/07 11:32:15 DEBUG : HTTP RESPONSE (req 0xc0007d0500)
2021/03/07 11:32:15 DEBUG : HTTP/1.1 404 Not Found
Content-Length: 208
Accept-Ranges: bytes
Connection: Keep-Alive
Content-Type: application/xml
Date: Sun, 07 Mar 2021 08:32:15 GMT
X-Amz-Request-Id: tx000000000000000c49331-0060448f8f-a8e1a64-prod

2021/03/07 11:32:15 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/07 11:32:15 DEBUG : b20175f4-ca2c-47e5-a2b5-24acc16c2b39: Need to transfer - File not found at Destination
2021/03/07 11:32:15 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/07 11:32:15 DEBUG : HTTP REQUEST (req 0xc00055c300)
2021/03/07 11:32:15 DEBUG : GET /abc/descriptor/2018/08/28/b20175f4-ca2c-47e5-a2b5-24acc16c2b39 HTTP/1.1
Host: 10.151.106.101:8091
User-Agent: rclone/v1.54.0-DEV
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210307T083215Z
Accept-Encoding: gzip

2021/03/07 11:32:15 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/07 11:32:15 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/07 11:32:15 DEBUG : HTTP RESPONSE (req 0xc00055c300)
2021/03/07 11:32:15 DEBUG : HTTP/1.1 200 OK
Content-Length: 86272
Accept-Ranges: bytes
Connection: Keep-Alive
Content-Type: application/octet-stream
Date: Sun, 07 Mar 2021 08:32:15 GMT
Etag: "3000ab75d49f8c0d64cd1b52fbbec5ef"
Last-Modified: Tue, 16 Jun 2020 10:16:59 GMT
X-Amz-Meta-Feature-Count: 674
X-Amz-Request-Id: tx000000000000000000100-0060448f8f-1afb26ba-ank
X-Rgw-Object-Type: Normal

2021/03/07 11:32:15 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/07 11:32:15 DEBUG : b20175f4-ca2c-47e5-a2b5-24acc16c2b39: src = &s3.Object{fs:(*s3.Fs)(0xc00052ba00), remote:"b20175f4-ca2c-47e5-a2b5-24acc16c2b39", etag:"\"3000ab75d49f8c0d64cd1b52fbbec5ef\"", bytes:86272, lastModified:time.Time{wall:0x0, ext:63727899419, loc:(*time.Location)(nil)}, meta:map[string]*string{"Feature-Count":(*string)(0xc0007c02d0)}, mimeType:"application/octet-stream", storageClass:""}
2021/03/07 11:32:15 DEBUG : b20175f4-ca2c-47e5-a2b5-24acc16c2b39: Reading metadata from b20175f4-ca2c-47e5-a2b5-24acc16c2b39
2021/03/07 11:32:15 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/07 11:32:15 DEBUG : HTTP REQUEST (req 0xc0007d0a00)
2021/03/07 11:32:15 DEBUG : PUT /abc/descriptor/2018/08/28/b20175f4-ca2c-47e5-a2b5-24acc16c2b39?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=A6T7AOXGXC9YD667UBJ2%2F20210307%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210307T083215Z&X-Amz-Expires=900&X-Amz-SignedHeaders=content-md5%3Bcontent-type%3Bhost%3Bx-amz-acl%3Bx-amz-meta-feature-count%3Bx-amz-meta-mtime&X-Amz-Signature=90ad8ac16dd3a392725aa1378cd528b4924aeef8a0ffc35d09d614b1549bc236 HTTP/1.1
Host: 10.151.106.172:8091
User-Agent: rclone/v1.54.0-DEV
Content-Length: 86272
content-md5: MACrddSfjA1kzRtS+77F7w==
content-type: application/octet-stream
x-amz-acl: private
x-amz-meta-feature-count: 674
x-amz-meta-mtime: 1592302619
Accept-Encoding: gzip

2021/03/07 11:32:15 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/07 11:32:15 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/07 11:32:15 DEBUG : HTTP RESPONSE (req 0xc0007d0a00)
2021/03/07 11:32:15 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Accept-Ranges: bytes
Connection: Keep-Alive
Date: Sun, 07 Mar 2021 08:32:15 GMT
Etag: "3000ab75d49f8c0d64cd1b52fbbec5ef"
X-Amz-Request-Id: tx000000000000000c49332-0060448f8f-a8e1a64-prod
X-Amz-Version-Id: SQNGTU19TqK4eqWGxxkqpf0H9RTZZUt

2021/03/07 11:32:15 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/07 11:32:15 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/07 11:32:15 DEBUG : HTTP REQUEST (req 0xc00055c700)
2021/03/07 11:32:15 DEBUG : HEAD /abc/descriptor/2018/08/28/b20175f4-ca2c-47e5-a2b5-24acc16c2b39 HTTP/1.1
Host: 10.151.106.172:8091
User-Agent: rclone/v1.54.0-DEV
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210307T083215Z

2021/03/07 11:32:15 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/03/07 11:32:15 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/07 11:32:15 DEBUG : HTTP RESPONSE (req 0xc00055c700)
2021/03/07 11:32:15 DEBUG : HTTP/1.1 200 OK
Content-Length: 86272
Accept-Ranges: bytes
Connection: Keep-Alive
Content-Type: application/octet-stream
Date: Sun, 07 Mar 2021 08:32:15 GMT
Etag: "3000ab75d49f8c0d64cd1b52fbbec5ef"
Last-Modified: Sun, 07 Mar 2021 08:32:15 GMT
X-Amz-Meta-Feature-Count: 674
X-Amz-Meta-Mtime: 1592302619
X-Amz-Request-Id: tx000000000000000c49333-0060448f8f-a8e1a64-prod
X-Amz-Version-Id: SQNGTU19TqK4eqWGxxkqpf0H9RTZZUt
X-Rgw-Object-Type: Normal

2021/03/07 11:32:15 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/03/07 11:32:15 DEBUG : b20175f4-ca2c-47e5-a2b5-24acc16c2b39: MD5 = 3000ab75d49f8c0d64cd1b52fbbec5ef OK
2021/03/07 11:32:15 INFO  : b20175f4-ca2c-47e5-a2b5-24acc16c2b39: Copied (new)
2021/03/07 11:32:15 INFO  :
Transferred:   	   84.250k / 84.250 kBytes, 100%, 5.807 MBytes/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:         0.1s

2021/03/07 11:32:15 DEBUG : 7 go routines active

Now the problem is I already have written 30M object. If I try "SYNC" to write only headers Rclone says they're identical.

2021/03/07 11:31:15 DEBUG : -16c2b39: Size and modification time the same (differ by 0s, within tolerance 1ns)
2021/03/07 11:31:15 DEBUG : -16c2b39: Unchanged skipping

1- Is there any easy way to write only headers?
2- Why you dont merge the metadata fix to master?
3- Do I need to be careful on any issue with fix-111 branch?

ivandeex · March 7, 2021, 9:30am

I remember I had an idea to setup rclone testbed using a dockerized ceph rados.
Do you think we want it in the integration test suite?
(to have more confidence in changes from such beta branches, for example)
If yes, I'll create a request and assign myself.
I don't promise it soon, just a "yes/no/maybe?" idea.

ncw · March 7, 2021, 11:56am

Thank you

Great

The code as it is at the moment doesn't sync metadata, it only copies it whenever the object changes.

There isn't an easy way to do this in the S3 protocol - you can't modify existing objects. So you have to make a copy of the object with new metadata.

Rclone could do this but it doesn't at the moment.

That is a good question...

I was hoping to make something a bit more general purpose - the fix in the 111 branch only copies metadata from s3 -> s3, but I haven't had time.

I'm also unsure about whether this introduces more HEAD requests when copying so might need to be enabled with a flag.

I rebased the patch to lastest rclone beta to pick up the latest fixes. Otherwise it should be fine!

v1.55.0-beta.5247.b7199fe3d.fix-111-metadata on branch fix-111-metadata (uploaded in 15-30 mins)

morphin · March 7, 2021, 2:36pm

Thanks for the new build. Now its faster somehow

Metadata flag will be awesome due to S3 depends on header infos like cleanup time etc.

system · May 7, 2021, 10:36am

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.