Hi.
I need to migrate the objects of ceph rgw cluster A to another cluster B，My question is, can I have multiple instances of sync（or copy） commads on a single bucket, each of which migrate a different set of objects. There are a large number of objects in out s3 buckets.
Thanks guys!
rclone version)rclone version is v1.53.3.
centos 7
hello and welcome to the forum,
yes, one bucket can be shared with muliple instances of sync.
sync each instance to a different folder inside that destination bucket.
a very simple example would be
rclone sync clustera:folder1 clusterb:folder1
rclone sync clustera:folder2 clusterb:folder2
but rclone has a lot of flags including filtering of what is copied
https://rclone.org/filtering/
and when testing use
https://rclone.org/docs/#n-dry-run
Yes that is fine.
I recommend you use the --checksum flag and --fast-list for maximum speed for s3 -> s3 copies.
for example： pc（centos） 1: sync clustera:folder1 clusterb:folder1
pc（centos） 2: sync clustera:folder1 clusterb:folder1
pc （centos） 3: sync clustera:folder1 clusterb:folder1
.......
The source address and destination address of the migration are the same ，if I run the rclone sync（or copy）again on the same s3 bucket, how can I make sure the objects in the s3 bucket are not uploaded twice?
for example： pc（centos） 1: sync clustera:folder1 clusterb:folder1
pc（centos） 2: sync clustera:folder1 clusterb:folder1
pc （centos） 3: sync clustera:folder1 clusterb:folder1
.......
The source address and destination address of the migration are the same ，if I run the rclone sync（or copy）again on the same s3 bucket, how can I make sure the objects in the s3 bucket are not uploaded twice?
if you run the same sync command over and over again, rclone only sync the changes, will not re-upload the same file twice.
"Sync the source to the destination, changing the destination only. Doesn't transfer unchanged files"
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.