What is the problem you are having with rclone?
Rclone is setting some bit of metadata on objects uploaded to IBM COS buckets from legacy block and file storage systems. This prevents end users from being able to use canned operations (rename) to manage content in applications such as cyberduck and filezilla pro. This also has implications in the s3fs fuse module's interpretation of the ownership of objects. Is there a way to have rclone uploads behave or look more like other methods of ingest? I'd prefer not to have to re-touch every object that is migrated in order for users to be able to manage their objects as expected.
This is part of a large scale migration effort to move storage from legacy block and file to object storage.
Example rClone upload (test env):
[myuser][l3mbp13]> aws s3api head-object --key RCLONE/CentOS-7-x86_64-NetInstall-2009.iso --endpoint-url https://s3.us.cloud-object-storage.appdomain.cloud --bucket rclone-testing-metadata-vs-api
bytes 602931200 application/x-iso9660-image "77c9dd4775432c811ae764984c56f955-18" Fri, 17 Dec 2021 15:37:53 GMT 18
METADATA DOjgZlXp/Su/n0eT+UCSXw== 1603729572
Example rClone upload (prod env)
[myuser][l3mbp13]> aws s3api head-object --key RCLONE-prod/CentOS-7-x86_64-NetInstall-2009.iso --endpoint-url https://s3.us.cloud-object-storage.appdomain.cloud --bucket rclone-testing-metadata-vs-api
bytes 602931200 application/octet-stream "77c9dd4775432c811ae764984c56f955-18" Fri, 17 Dec 2021 15:55:29 GMT 18
METADATA DOjgZlXp/Su/n0eT+UCSXw== 1603729572
Example AWS CLI upload:
[myuser][l3mbp13]> aws s3api head-object --key AWSCLI/CentOS-7-x86_64-NetInstall-2009.iso --endpoint-url https://s3.us.cloud-object-storage.appdomain.cloud --bucket rclone-testing-metadata-vs-api
bytes 602931200 application/x-iso9660-image "5d57834651dee45ebf25e9df0b74c4e1-72" Fri, 17 Dec 2021 14:59:33 GMT 72
What is your rclone version (output from rclone version
)
Production env:
rclone v1.48.0
- os/arch: linux/amd64
- go version: go1.12.6
Test Env:
rclone v1.55.1
- os/type: darwin
- os/arch: amd64
- go/version: go1.16.3
- go/linking: dynamic
- go/tags: none
Which cloud storage system are you using? (eg Google Drive)
IBM Cloud Object Storage
The command you were trying to run (eg rclone copy /tmp remote:tmp
)
rclone copy /Users/myuser/Desktop/upload-metadata-testing/RCLONE/ metadata-testing:rclone-testing-metadata-vs-api/rclone/ --log-file=/Users/myuser/rclone-test.log -vP --s3-upload-concurrency=16 --s3-chunk-size=32M --transfers=100
The rclone config contents with secrets removed.
[metadata-testing]
type = s3
provider = IBMCOS
env_auth = false
access_key_id = <xxx>
secret_access_key = <xxx>
endpoint = s3.us.cloud-object-storage.appdomain.cloud
A log from the command with the -vv
flag
2021/12/17 09:35:35 DEBUG : 8 go routines active
2021/12/17 09:37:12 DEBUG : Using config file from "/var/root/.config/rclone/rclone.conf"
2021/12/17 09:37:12 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "copy" "/Users/myuser/Desktop/upload-metadata-testing/RCLONE/" "metadata-testing:rclone-testing-metadata-vs-api/RCLONE/" "--log-file=/Users/myuser/rclone-test.log" "-vvP" "--s3-upload-concurrency=16" "--s3-chunk-size=32M" "--transfers=100"]
2021/12/17 09:37:12 DEBUG : Creating backend with remote "/Users/myuser/Desktop/upload-metadata-testing/RCLONE/"
2021/12/17 09:37:12 DEBUG : Creating backend with remote "metadata-testing:rclone-testing-metadata-vs-api/RCLONE/"
2021/12/17 09:37:12 DEBUG : metadata-testing: detected overridden config - adding "{Gv6vy}" suffix to name
2021/12/17 09:37:12 DEBUG : fs cache: renaming cache item "metadata-testing:rclone-testing-metadata-vs-api/RCLONE/" to be canonical "metadata-testing{Gv6vy}:rclone-testing-metadata-vs-api/RCLONE"
2021/12/17 09:37:13 DEBUG : S3 bucket rclone-testing-metadata-vs-api path RCLONE: Waiting for checks to finish
2021/12/17 09:37:13 DEBUG : S3 bucket rclone-testing-metadata-vs-api path RCLONE: Waiting for transfers to finish
2021/12/17 09:37:13 DEBUG : .DS_Store: MD5 = 194577a7e20bdcc7afbb718f502c134c OK
2021/12/17 09:37:13 INFO : .DS_Store: Copied (new)
2021/12/17 09:37:16 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 1 size 32M offset 0/575M
2021/12/17 09:37:16 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 2 size 32M offset 32M/575M
2021/12/17 09:37:16 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 3 size 32M offset 64M/575M
2021/12/17 09:37:16 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 4 size 32M offset 96M/575M
2021/12/17 09:37:16 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 5 size 32M offset 128M/575M
2021/12/17 09:37:16 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 6 size 32M offset 160M/575M
2021/12/17 09:37:16 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 7 size 32M offset 192M/575M
2021/12/17 09:37:17 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 8 size 32M offset 224M/575M
2021/12/17 09:37:17 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 9 size 32M offset 256M/575M
2021/12/17 09:37:17 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 10 size 32M offset 288M/575M
2021/12/17 09:37:17 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 11 size 32M offset 320M/575M
2021/12/17 09:37:17 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 12 size 32M offset 352M/575M
2021/12/17 09:37:17 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 13 size 32M offset 384M/575M
2021/12/17 09:37:18 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 14 size 32M offset 416M/575M
2021/12/17 09:37:18 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 15 size 32M offset 448M/575M
2021/12/17 09:37:18 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 16 size 32M offset 480M/575M
2021/12/17 09:37:39 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 17 size 32M offset 512M/575M
2021/12/17 09:37:40 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: multipart upload starting chunk 18 size 31M offset 544M/575M
2021/12/17 09:37:54 DEBUG : CentOS-7-x86_64-NetInstall-2009.iso: MD5 = 0ce8e06655e9fd2bbf9f4793f940925f OK
2021/12/17 09:37:54 INFO : CentOS-7-x86_64-NetInstall-2009.iso: Copied (new)
2021/12/17 09:37:54 INFO :
Transferred: 575.006M / 575.006 MBytes, 100%, 13.968 MBytes/s, ETA 0s
Transferred: 2 / 2, 100%
Elapsed time: 42.4s
2021/12/17 09:37:54 DEBUG : 37 go routines active