Rclone sync from "other provider" to aws s3 : personnal metadata are lost

What is the problem you are having with rclone?

I try to use rsync to "backup" my local object storage platform to AWS.
My local object storage platform is Scality Ring and it provides S3 connector/API.

I'm able to synchronize the data but I lost my personnal metadata in AWS.
Example of a metadata: x-amz-meta-filename
All my personnal metadata have the "x-amz-meta-" prefix.

But if I try from AWS to AWS it's ok: my metadata are still present in the targeted bucket.

I tried to cheat by changing my rclone configuration as:

  • before: provider = Other
  • after: provider = AWS

But the problem still persists...

It's not a problem with rights because with "aws cli" I'm able to retrieve the metadata with "head-object" command using the same access key / private key.

What is your rclone version (output from rclone version)

I tried with rclone 1.47 (provide by CentOS 7) and 1.51 (downloaded from rclone.org).

Which OS you are using and how many bits (eg Windows 7, 64 bit)

CentOS 7.6.1810 64 bits.

Which cloud storage system are you using? (eg Google Drive)

Source : Scality Ring
Target : Amazon S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

I tried rclone sync:
time rclone --progress sync LOCAL_S3:mysourcebucket REMOTE_AWS:mytargetbucket

And rclone copy:
time rclone --progress sync LOCAL_S3:mysourcebucket REMOTE_AWS:mytargetbucket

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

time rclone -vv --progress sync LOCAL_S3:mysourcebucket REMOTE_AWS:mytargetbucket
2020/04/23 15:56:25 DEBUG : rclone: Version "v1.51.0" starting with parameters ["./rclone" "-vv" "--progress" "sync" "LOCAL_S3:mysourcebucket" "REMOTE_AWS:mytargetbucket"]
2020/04/23 15:56:25 DEBUG : Using config file from "/home/kortex/.config/rclone/rclone.conf"
2020-04-23 15:56:25 INFO : S3 bucket mytargetbucket: Waiting for checks to finish
2020-04-23 15:56:25 INFO : S3 bucket mytargetbucket: Waiting for transfers to finish
2020-04-23 15:56:25 DEBUG : 0000000000000000000000002: MD5 = 41209059697e7ec10c829d8d4d5616aa OK
2020-04-23 15:56:25 INFO : 0000000000000000000000002: Copied (new)
2020-04-23 15:56:25 DEBUG : 0000000000000000000000003: MD5 = 41209059697e7ec10c829d8d4d5616aa OK
2020-04-23 15:56:25 INFO : 0000000000000000000000003: Copied (new)
2020-04-23 15:56:25 INFO : Waiting for deletions to finish
Transferred: 1.435M / 1.435 MBytes, 100%, 4.325 MBytes/s, ETA 0s
Transferred: 2 / 2, 100%
Elapsed time: 0.3s
2020/04/23 15:56:25 INFO :
Transferred: 1.435M / 1.435 MBytes, 100%, 4.325 MBytes/s, ETA 0s
Transferred: 2 / 2, 100%
Elapsed time: 0.3s

2020/04/23 15:56:25 DEBUG : 15 go routines active
2020/04/23 15:56:25 DEBUG : rclone: Version "v1.51.0" finishing with parameters ["./rclone" "-vv" "--progress" "sync" "LOCAL_S3:mysourcebucket" "REMOTE_AWS:mytargetbucket"]

real 0m0.475s
user 0m0.060s
sys 0m0.023s

Anyone has an idea please :slight_smile: ?

I have a feeling this got fixed... Can you try with the latest release?

I have already tried with the 1.51.0.
I have tried with :

  • 1.47 (centos package)
  • 1.51.0

No, sorry, I imagined that! There have been discussions about that in the past and I got confused.

There is an issue about it open

We'll preserve the metadata if you do a server side copy, but not otherwise.

Fancy helping out with the issue?

Hi ncw,

How can I help :slight_smile: ?
I'm honest I don't have tried but I'm not sure to be able to develop the feature myself in Go...

I have read quickly the discussion in the Github feature request: my need is similar but the final case is different.

In the Github discussion, there is a discussion about the difficulty to translate the metadata from one technology (example: Amazon AWS) to another technology (example: swift): I can easily imagine the difficulty :confused:

In my case, I think it's easier (I'm not sure: that's why I say "I think"):

  • my source is a software that are compatible with S3 protocol (not all the protocol obviously but it covers a good part)
  • my source software support metadata in the same format than AWS (x-amz-meta-)

In a first step, why not to implement the feature for the synchronization between compatible systems (S3 -> S3, Swift -> Swift, etc)?
For example, by adding a parameter when we configure a profil to say "I want sync metadata".
And if the provider of the solution is not compatible and/or if one of the two profils is a differente technology (S3 -> Swift, etc), too bad: the user will disable it and the parameter will be ignored.

1 Like

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.