Rclone setting object metadata s3/IBM COS

What is the problem you are having with rclone?

Rclone is setting some bit of metadata on objects uploaded to IBM COS buckets from legacy block and file storage systems. This prevents end users from being able to use canned operations (rename) to manage content in applications such as cyberduck and filezilla pro. This also has implications in the s3fs fuse module's interpretation of the ownership of objects. Is there a way to have rclone uploads behave or look more like other methods of ingest? I'd prefer not to have to re-touch every object that is migrated in order for users to be able to manage their objects as expected.

This is part of a large scale migration effort to move storage from legacy block and file to object storage.

Example rClone upload (test env):

[myuser][l3mbp13]> aws s3api head-object --key RCLONE/CentOS-7-x86_64-NetInstall-2009.iso  --endpoint-url https://s3.us.cloud-object-storage.appdomain.cloud --bucket rclone-testing-metadata-vs-api
bytes	602931200	application/x-iso9660-image	"77c9dd4775432c811ae764984c56f955-18"	Fri, 17 Dec 2021 15:37:53 GMT	18
METADATA	DOjgZlXp/Su/n0eT+UCSXw==	1603729572

Example rClone upload (prod env)

[myuser][l3mbp13]> aws s3api head-object --key RCLONE-prod/CentOS-7-x86_64-NetInstall-2009.iso  --endpoint-url https://s3.us.cloud-object-storage.appdomain.cloud --bucket rclone-testing-metadata-vs-api
bytes	602931200	application/octet-stream	"77c9dd4775432c811ae764984c56f955-18"	Fri, 17 Dec 2021 15:55:29 GMT	18
METADATA	DOjgZlXp/Su/n0eT+UCSXw==	1603729572

Example AWS CLI upload:

[myuser][l3mbp13]> aws s3api head-object --key AWSCLI/CentOS-7-x86_64-NetInstall-2009.iso  --endpoint-url https://s3.us.cloud-object-storage.appdomain.cloud --bucket rclone-testing-metadata-vs-api
bytes	602931200	application/x-iso9660-image	"5d57834651dee45ebf25e9df0b74c4e1-72"	Fri, 17 Dec 2021 14:59:33 GMT	72

What is your rclone version (output from rclone version)

Production env:

rclone v1.48.0
- os/arch: linux/amd64
- go version: go1.12.6

Test Env:

rclone v1.55.1
- os/type: darwin
- os/arch: amd64
- go/version: go1.16.3
- go/linking: dynamic
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

IBM Cloud Object Storage

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy /Users/myuser/Desktop/upload-metadata-testing/RCLONE/ metadata-testing:rclone-testing-metadata-vs-api/rclone/ --log-file=/Users/myuser/rclone-test.log -vP --s3-upload-concurrency=16 --s3-chunk-size=32M --transfers=100

The rclone config contents with secrets removed.

[metadata-testing]
type = s3
provider = IBMCOS
env_auth = false
access_key_id = <xxx>
secret_access_key = <xxx>
endpoint = s3.us.cloud-object-storage.appdomain.cloud

A log from the command with the -vv flag

2021/12/17 09:35:35 DEBUG : 8 go routines active
2021/12/17 09:37:12 DEBUG : Using config file from "/var/root/.config/rclone/rclone.conf"
2021/12/17 09:37:12 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "copy" "/Users/myuser/Desktop/upload-metadata-testing/RCLONE/" "metadata-testing:rclone-testing-metadata-vs-api/RCLONE/" "--log-file=/Users/myuser/rclone-test.log" "-vvP" "--s3-upload-concurrency=16" "--s3-chunk-size=32M" "--transfers=100"]
2021/12/17 09:37:12 DEBUG : Creating backend with remote "/Users/myuser/Desktop/upload-metadata-testing/RCLONE/"
2021/12/17 09:37:12 DEBUG : Creating backend with remote "metadata-testing:rclone-testing-metadata-vs-api/RCLONE/"
2021/12/17 09:37:12 DEBUG : metadata-testing: detected overridden config - adding "{Gv6vy}" suffix to name
2021/12/17 09:37:12 DEBUG : fs cache: renaming cache item "metadata-testing:rclone-testing-metadata-vs-api/RCLONE/" to be canonical "metadata-testing{Gv6vy}:rclone-testing-metadata-vs-api/RCLONE"
2021/12/17 09:37:13 DEBUG : S3 bucket rclone-testing-metadata-vs-api path RCLONE: Waiting for checks to finish
2021/12/17 09:37:13 DEBUG : S3 bucket rclone-testing-metadata-vs-api path RCLONE: Waiting for transfers to finish
2021/12/17 09:37:13 DEBUG : .DS_Store: MD5 = 194577a7e20bdcc7afbb718f502c134c OK
2021/12/17 09:37:13 INFO  : .DS_Store: Copied (new)
2021/12/17 09:37:16 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 1 size 32M offset 0/575M
2021/12/17 09:37:16 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 2 size 32M offset 32M/575M
2021/12/17 09:37:16 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 3 size 32M offset 64M/575M
2021/12/17 09:37:16 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 4 size 32M offset 96M/575M
2021/12/17 09:37:16 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 5 size 32M offset 128M/575M
2021/12/17 09:37:16 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 6 size 32M offset 160M/575M
2021/12/17 09:37:16 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 7 size 32M offset 192M/575M
2021/12/17 09:37:17 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 8 size 32M offset 224M/575M
2021/12/17 09:37:17 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 9 size 32M offset 256M/575M
2021/12/17 09:37:17 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 10 size 32M offset 288M/575M
2021/12/17 09:37:17 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 11 size 32M offset 320M/575M
2021/12/17 09:37:17 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 12 size 32M offset 352M/575M
2021/12/17 09:37:17 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 13 size 32M offset 384M/575M
2021/12/17 09:37:18 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 14 size 32M offset 416M/575M
2021/12/17 09:37:18 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 15 size 32M offset 448M/575M
2021/12/17 09:37:18 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 16 size 32M offset 480M/575M
2021/12/17 09:37:39 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 17 size 32M offset 512M/575M
2021/12/17 09:37:40 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 18 size 31M offset 544M/575M
2021/12/17 09:37:54 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: MD5 = 0ce8e06655e9fd2bbf9f4793f940925f OK
2021/12/17 09:37:54 INFO  : CentOS-7-x86_64-NetInstall-2009.iso: Copied (new)
2021/12/17 09:37:54 INFO  :
Transferred:   	  575.006M / 575.006 MBytes, 100%, 13.968 MBytes/s, ETA 0s
Transferred:            2 / 2, 100%
Elapsed time:        42.4s

2021/12/17 09:37:54 DEBUG : 37 go routines active

Those are old versions. Can you test with the latest please?

New version command:

l3mbp13:~ root# /Users/myuser/Downloads/rclone-v1.57.0-osx-amd64/rclone copy /Users/myuser/Desktop/upload-metadata-testing/RCLONE metadata-testing:rclone-testing-metadata-vs-api/rclone-new-version/ --log-file=/Users/myuser/rclone-new-test.log -vvP --s3-upload-concurrency=16 --s3-chunk-size=32M --transfers=100
Transferred:   	  575.006 MiB / 575.006 MiB, 100%, 13.650 MiB/s, ETA 0s
Transferred:            2 / 2, 100%
Elapsed time:        32.1s

New version upload result:

[myuser][l3mbp13]> aws s3api head-object --key rclone-new-version/CentOS-7-x86_64-NetInstall-2009.iso  --endpoint-url https://s3.us.cloud-object-storage.appdomain.cloud --bucket rclone-testing-metadata-vs-api
bytes	602931200	application/x-cd-image	"77c9dd4775432c811ae764984c56f955-18"	Fri, 17 Dec 2021 16:16:33 GMT	18
METADATA	DOjgZlXp/Su/n0eT+UCSXw==	1603729572

Version:

[10:10][~/Downloads/rclone-v1.57.0-osx-amd64]
[myuser][l3mbp13]> ./rclone --version
rclone v1.57.0
- os/version: darwin 11.6.1 (64 bit)
- os/kernel: 20.6.0 (x86_64)
- os/type: darwin
- os/arch: amd64
- go/version: go1.17.2
- go/linking: dynamic
- go/tags: cmount

New log

2021/12/17 10:16:02 DEBUG : rclone: Version "v1.57.0" starting with parameters ["/Users/myuser/Downloads/rclone-v1.57.0-osx-amd64/rclone" "copy" "/Users/myuser/Desktop/upload-metadata-testing/RCLONE" "metadata-testing:rclone-testing-metadata-vs-api/rclone-new-version/" "--log-file=/Users/myuser/rclone-new-test.log" "-vvP" "--s3-upload-concurrency=16" "--s3-chunk-size=32M" "--transfers=100"]
2021/12/17 10:16:02 DEBUG : Creating backend with remote "/Users/myuser/Desktop/upload-metadata-testing/RCLONE"
2021/12/17 10:16:02 DEBUG : Using config file from "/var/root/.config/rclone/rclone.conf"
2021/12/17 10:16:02 DEBUG : Creating backend with remote "metadata-testing:rclone-testing-metadata-vs-api/rclone-new-version/"
2021/12/17 10:16:02 DEBUG : metadata-testing: detected overridden config - adding "{naLai}" suffix to name
2021/12/17 10:16:02 DEBUG : fs cache: renaming cache item "metadata-testing:rclone-testing-metadata-vs-api/rclone-new-version/" to be canonical "metadata-testing{naLai}:rclone-testing-metadata-vs-api/rclone-new-version"
2021/12/17 10:16:03 DEBUG : S3 bucket rclone-testing-metadata-vs-api path rclone-new-version: Waiting for checks to finish
2021/12/17 10:16:03 DEBUG : S3 bucket rclone-testing-metadata-vs-api path rclone-new-version: Waiting for transfers to finish
2021/12/17 10:16:04 DEBUG : .DS_Store: md5 = 194577a7e20bdcc7afbb718f502c134c OK
2021/12/17 10:16:04 INFO  : .DS_Store: Copied (new)
2021/12/17 10:16:06 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 1 size 32Mi offset 0/575Mi
2021/12/17 10:16:06 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 2 size 32Mi offset 32Mi/575Mi
2021/12/17 10:16:06 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 3 size 32Mi offset 64Mi/575Mi
2021/12/17 10:16:06 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 4 size 32Mi offset 96Mi/575Mi
2021/12/17 10:16:06 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 5 size 32Mi offset 128Mi/575Mi
2021/12/17 10:16:06 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 6 size 32Mi offset 160Mi/575Mi
2021/12/17 10:16:06 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 7 size 32Mi offset 192Mi/575Mi
2021/12/17 10:16:07 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 8 size 32Mi offset 224Mi/575Mi
2021/12/17 10:16:07 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 9 size 32Mi offset 256Mi/575Mi
2021/12/17 10:16:07 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 10 size 32Mi offset 288Mi/575Mi
2021/12/17 10:16:07 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 11 size 32Mi offset 320Mi/575Mi
2021/12/17 10:16:07 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 12 size 32Mi offset 352Mi/575Mi
2021/12/17 10:16:08 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 13 size 32Mi offset 384Mi/575Mi
2021/12/17 10:16:08 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 14 size 32Mi offset 416Mi/575Mi
2021/12/17 10:16:08 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 15 size 32Mi offset 448Mi/575Mi
2021/12/17 10:16:08 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 16 size 32Mi offset 480Mi/575Mi
2021/12/17 10:16:12 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 17 size 32Mi offset 512Mi/575Mi
2021/12/17 10:16:14 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 18 size 31Mi offset 544Mi/575Mi
2021/12/17 10:16:34 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: md5 = 0ce8e06655e9fd2bbf9f4793f940925f OK
2021/12/17 10:16:34 INFO  : CentOS-7-x86_64-NetInstall-2009.iso: Copied (new)
2021/12/17 10:16:34 INFO  :
Transferred:   	  575.006 MiB / 575.006 MiB, 100%, 13.650 MiB/s, ETA 0s
Transferred:            2 / 2, 100%
Elapsed time:        32.1s

2021/12/17 10:16:34 DEBUG : 38 go routines active

Same config file

Thanks.

I think all comes from the Amazon S3 SDK if I'm not mistaken so might be something unique with this S3 backend.

I think @ncw would need to comment as I don't use S3 at all.

hello and welcome to the forum,

by default, rclone creates metadata, such as md5 hash

x-amz-meta-md5chksum: j2gVFkX16+MRBaGLn5pVxg==

that can be disabled with --s3-disable-checksum

Is that METADATA tag though the one you mentioned? I wasn't sure how to correlate if it's grabbing that or was it something else.

here is the same file uploaded with and without --s3-disable-checksum

I'm trying to figure out / connect how that above connects with your output. What is being shown from that to correlate with the graphical output you've provided?

yeah, confusing, as in the OP example, the header names are missing.

so i would guesstimate

DOjgZlXp/Su/n0eT+UCSXw== is x-amz-meta-md5chksum

edit: did a quick test

aws s3api head-object --bucket zork --key file01.file --endpoint-url=https://s3.us-east-2.wasabisys.com 
{
    "AcceptRanges": "bytes",
    "LastModified": "2021-12-17T16:54:18+00:00",
    "ContentLength": 19772672,
    "ETag": "\"466757ae3890a74dcf457e8402f5240d-1\"",
    "ContentType": "application/octet-stream",
    "Metadata": {
        "md5chksum": "KBjKF4ZJgHA52toY8El+ow==",
        "mtime": "1628703435.50162"
    }
}
1 Like

and i would guesstimate, that the OP case,
1603729572 is modtime.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.