S3 minio bucket syncing with metadata

What is the problem you are having with rclone?

I am syncing data between two MinIO bucket on different remote. I found the migration for metadata confusing.

Syncing between two remote, none of the custom metadata is preserved. Besides, rclone adds additional X-Amz-Meta-Mtime to store mod-time (This is what I expect). On the other hand, syncing two buckets on same remote preserve custom metadata but no additional element for mod-time.

After all, my requirement is to sync between two remote (both minio) and preserve custom metadata as well. I recognized it is not available for different types of storage from other post and the question is does it support between same type of storage.

What is your rclone version (output from rclone version)

rclone v1.53.1

Which cloud storage system are you using? (eg Google Drive)

S3 MinIO

Hmm there is an issue about this

However I did implement this feature, I didn't release it yet though for some reason.

This doesn't need a flag to enable it - maybe it should as it potentially has to do more work (a HEAD request on each object before copying it).

Can you try this

v1.54.0-beta.4773.1528c78d0.fix-111-metadata on branch fix-111-metadata (uploaded in 15-30 mins)

1 Like

It works! Thanks a lots!
Syncing to an empty bucket keeps my custom metadata and also the mod-time. While syncing to a bucket containing objects without custom metadata does not transfer again. It is quite reasonable as it only checks size and modification time. However, this means that I will have to delete everything and do the whole syncing again :frowning: Is there any suggestion for this condition? Only some of the objects contain custom metadata so I did not notice at first. There are about 45TB or 200,000+ objects.

Yes, it doesn't update metadata...

If you could get a list of objects without the correct metadata then you could upload them again with --files-from-raw

Rclone can't sync metadata yes, alas.

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.