Syncing and updating metadata between endpoints

Feature
1

Hi,
I have red several topics here on the forum regarding metadata transfer. It was a partial success.
With our version 1.57 we successfully synced files with metadata from one bucket to another within the same endpoint (s3.us-south.cloud-object-storage.appdomain.cloud). However moving to 3rd bucket in the other endpoint (s3.us.cloud-object-storage.appdomain.cloud) did not propage the metadata.
Therefore as suggested we used a beta version with metadata fix (v1.55.0-beta.5247.b7199fe3d.fix-111-metadata).
As follow up problem we realized the meta data is not updated when files are synced between bucket. As a workaround we could delete the files and copy them over again but that defeats whole purpose of rclone sync.

How feasible would be to add metadata synchronization to sync command on S3 ?
Would having all bucket in same endpoint/location help of not needing to use the beta with the 111 fix ?
Are there other options to synchronize metadata with sync command ?

1, rclone sync --no-update-modtime COS_US_SOUTH:path1 COS_US:path2 -v
2,
3,
[COS_US_SOUTH]
type = s3
provider = IBMCOS
env_auth = false
access_key_id = ****
secret_access_key = ****
endpoint = s3.us-south.cloud-object-storage.appdomain.cloud

[COS_US]
type = s3
provider = IBMCOS
env_auth = false
access_key_id = ****
secret_access_key = ****
endpoint = s3.us.cloud-object-storage.appdomain.cloud

4, v1.55.0-beta.5247.b7199fe3d.fix-111-metadata

2

In the latest beta we've introduced the --metadata flag which will enable full metadata syncing between s3 backends. This will shortly become 1.59 so you should use this for your tests.

The metadata is copied when the file is uploaded only. There isn't (yet) a way to sync metadata on already existing objects.

1 Like
3

Thanks for speedy answer. I will definitelly test the new version.

Thou what are the chances to have sync metadata on already existing objects with rclone?

4

Great

Its not an impossible thing to ask for, but isn't on the road map at the moment as we need to let the metadata framework bed in, and widen the support to other backends.

5

Hello,

For a completnes and perhaps it will help other people.

We have tested rclone version
v1.60.0-beta.6355.67fd60275

and used command
rclone sync --ignore-times --metadata --no-update-modtime COS_US_SOUTH:source_path COS_US:target_path -v

I can confirm that it updates the metadata even if the file contents/object did not changed. Which is our use case and expected result.
--metadata assure the meta data is transferred
--ignore-times make sure the file is synced even if it was not changed
--no-update-modtime might not be neccessary we use it to keep the mod times while syncing objects between buckets

Thanks for the updates @ncw.

1 Like
6

Thanks for trying out the new feature.

7

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.