Need to sync S3 buckets with all ACL

Hello world :smile:

What is the problem you are having with rclone?

I need to sync S3 buckets with all ACL beetween CEPH and minIO. All objects have different type of ACL. I need to sync like a mirror and restore some object and specific ACL from minIO to CEPH if asked by customer. Problem, when I delete all objects from CEPH and sync all objects from minIO to retore them, objects are restored but I lost all ACL.

What is your rclone version (output from rclone version)

/usr/local/bin/rclone --version
rclone v1.51.0

  • os/arch: linux/amd64
  • go version: go1.13.7

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Linux CentOS 7.8 64-bits

Which cloud storage system are you using? (eg Google Drive)

On-premise CEPH & minIO

The command you were trying to run (eg rclone copy /tmp remote:tmp)

$ /usr/local/bin/rclone -v sync CephS3:/blog-s3-dev minIO:/blog-s3-dev 

$ /usr/local/bin/rclone -v --s3-bucket-acl=bucket-owner-full-control sync CephS3:/blog-s3-dev minIO:/blog-s3-dev 

The rclone config contents with secrets removed.

[CephS3]
type = s3
provider = Ceph
env_auth = false
access_key_id = XXXXXXXXXXXXXXXX
secret_access_key = XXXXXXXXXXXXXXXXXXXXXXXX
endpoint = cephs3.foo.com

[minIO]
type = s3
provider = Minio
env_auth = false
access_key_id = XXXXXXXXXXXX
secret_access_key = XXXXXXXXXXXXXXXXXXXXXXXXXXX
endpoint = http://10.0.0.1:9000
acl = bucket-owner-full-control

A log from the command with the -vv flag

/usr/local/bin/rclone -vv --s3-bucket-acl=bucket-owner-full-control sync minIO:/blog-s3-dev CephS3:/blog-s3-dev

2020/07/22 21:27:34 DEBUG : rclone: Version "v1.51.0" starting with parameters ["/usr/local/bin/rclone" "-vv" "--s3-bucket-acl=bucket-owner-full-control" "sync" "minIO:/blog-s3-dev" "CephS3:/blog-s3-dev"]
2020/07/22 21:27:34 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2020/07/22 21:27:35 INFO  : S3 bucket blog-s3-dev: Waiting for checks to finish
2020/07/22 21:27:35 DEBUG : index.html: Size and modification time the same (differ by 0s, within tolerance 1ns)
2020/07/22 21:27:35 DEBUG : index.html: Unchanged skipping
2020/07/22 21:27:35 DEBUG : Ceph_Logo.png: Size and modification time the same (differ by 0s, within tolerance 1ns)
2020/07/22 21:27:35 DEBUG : Ceph_Logo.png: Unchanged skipping
2020/07/22 21:27:35 INFO  : S3 bucket blog-s3-dev: Waiting for transfers to finish
2020/07/22 21:27:35 INFO  : Waiting for deletions to finish
2020/07/22 21:27:35 INFO  :
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:                 2 / 2, 100%
Elapsed time:         0.0s

2020/07/22 21:27:35 DEBUG : 11 go routines active
2020/07/22 21:27:35 DEBUG : rclone: Version "v1.51.0" finishing with parameters ["/usr/local/bin/rclone" "-vv" "--s3-bucket-acl=bucket-owner-full-control" "sync" "minIO:/blog-s3-dev" "CephS3:/blog-s3-dev"]

Thank you for reply.
Mirzouch

How many different ACLs are there? Maybe you could do one sync for each type of ACL?

I'm not sure how you list which ACL objects have - what tool do you use for that?

Hello Nick,
Thank you for reply. We can use a python script to identify or backup ACL of objects. But in fact I thought that rclone could do it natively, I may have misunderstood the man.

Could you confirm that rclone cannot sync object with is ACL, please.

Thank you and take care
Mirzouch

You can set the ACL of objects when you create them, but rclone can't read the ACL from existing objects and preserve it when you create new objects.

1 Like

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.