What is the problem you are having with rclone?
I try to use rsync to "backup" my local object storage platform to AWS.
My local object storage platform is Scality Ring and it provides S3 connector/API.
I'm able to synchronize the data but I lost my personnal metadata in AWS.
Example of a metadata: x-amz-meta-filename
All my personnal metadata have the "x-amz-meta-" prefix.
But if I try from AWS to AWS it's ok: my metadata are still present in the targeted bucket.
I tried to cheat by changing my rclone configuration as:
- before: provider = Other
- after: provider = AWS
But the problem still persists...
It's not a problem with rights because with "aws cli" I'm able to retrieve the metadata with "head-object" command using the same access key / private key.
What is your rclone version (output from rclone version
)
I tried with rclone 1.47 (provide by CentOS 7) and 1.51 (downloaded from rclone.org).
Which OS you are using and how many bits (eg Windows 7, 64 bit)
CentOS 7.6.1810 64 bits.
Which cloud storage system are you using? (eg Google Drive)
Source : Scality Ring
Target : Amazon S3
The command you were trying to run (eg rclone copy /tmp remote:tmp
)
I tried rclone sync:
time rclone --progress sync LOCAL_S3:mysourcebucket REMOTE_AWS:mytargetbucket
And rclone copy:
time rclone --progress sync LOCAL_S3:mysourcebucket REMOTE_AWS:mytargetbucket
A log from the command with the -vv
flag (eg output from rclone -vv copy /tmp remote:tmp
)
time rclone -vv --progress sync LOCAL_S3:mysourcebucket REMOTE_AWS:mytargetbucket
2020/04/23 15:56:25 DEBUG : rclone: Version "v1.51.0" starting with parameters ["./rclone" "-vv" "--progress" "sync" "LOCAL_S3:mysourcebucket" "REMOTE_AWS:mytargetbucket"]
2020/04/23 15:56:25 DEBUG : Using config file from "/home/kortex/.config/rclone/rclone.conf"
2020-04-23 15:56:25 INFO : S3 bucket mytargetbucket: Waiting for checks to finish
2020-04-23 15:56:25 INFO : S3 bucket mytargetbucket: Waiting for transfers to finish
2020-04-23 15:56:25 DEBUG : 0000000000000000000000002: MD5 = 41209059697e7ec10c829d8d4d5616aa OK
2020-04-23 15:56:25 INFO : 0000000000000000000000002: Copied (new)
2020-04-23 15:56:25 DEBUG : 0000000000000000000000003: MD5 = 41209059697e7ec10c829d8d4d5616aa OK
2020-04-23 15:56:25 INFO : 0000000000000000000000003: Copied (new)
2020-04-23 15:56:25 INFO : Waiting for deletions to finish
Transferred: 1.435M / 1.435 MBytes, 100%, 4.325 MBytes/s, ETA 0s
Transferred: 2 / 2, 100%
Elapsed time: 0.3s
2020/04/23 15:56:25 INFO :
Transferred: 1.435M / 1.435 MBytes, 100%, 4.325 MBytes/s, ETA 0s
Transferred: 2 / 2, 100%
Elapsed time: 0.3s
2020/04/23 15:56:25 DEBUG : 15 go routines active
2020/04/23 15:56:25 DEBUG : rclone: Version "v1.51.0" finishing with parameters ["./rclone" "-vv" "--progress" "sync" "LOCAL_S3:mysourcebucket" "REMOTE_AWS:mytargetbucket"]
real 0m0.475s
user 0m0.060s
sys 0m0.023s
Anyone has an idea please ?