Sync and last modified

Help and Support
1

What is the problem you are having with rclone?

Tôi đang thử nghiệm đồng bộ dữ liệu từ hai hệ thống S3 là Ceph và Cloudian, tôi có thể đồng bộ dữ liệu giữa chúng nhưng tôi thấy trường "sửa đổi lần cuối" không được đồng bộ, Cloudian chỉ hiển thị thời gian tôi đã chạy câu lệnh đồng bộ.

Run the command 'rclone version' and share the full output of the command.

rclone v1.67.0
- os/version: centos 7.9.2009 (64 bit)
- os/kernel: 3.10.0-1160.119.1.el7.x86_64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.22.4
- go/linking: static
- go/tags: none

Which cloud storage system are you using?

ceph, cloudian

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync ceph:ceph2-0 cloudian:ceph2-0 --progress --transfers=2 --buffer-size=32M --metadata --refresh-times

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

 rclone config redacted
[ceph]
type = s3
provider = Other
access_key_id = XXX
secret_access_key = XXX
region = other-v2-signature
endpoint = http://10.200.10.163:8080
acl = bucket-owner-full-control

[cloudian]
type = s3
provider = Other
access_key_id = XXX
secret_access_key = XXX
region = HN
endpoint = http://s3-hn.trucnv.local:80
acl = bucket-owner-full-control
bucket_acl = authenticated-read
### Double check the config for sensitive info before posting publicly

A log from the command that you were trying to run with the -vv flag

 rclone sync ceph:ceph2-0 cloudian:ceph2-0 --progress --transfers=2 --buffer-size=32M --metadata --refresh-times -vv
2024/08/29 15:41:28 DEBUG : rclone: Version "v1.67.0" starting with parameters ["rclone" "sync" "ceph:ceph2-0" "cloudian:ceph2-0" "--progress" "--transfers=2" "--buffer-size=32M" "--metadata" "--refresh-times" "-vv"]
2024/08/29 15:41:28 DEBUG : Creating backend with remote "ceph:ceph2-0"
2024/08/29 15:41:28 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2024/08/29 15:41:28 DEBUG : Resolving service "s3" region "other-v2-signature"
2024/08/29 15:41:28 DEBUG : Using v2 auth
2024/08/29 15:41:28 DEBUG : Creating backend with remote "cloudian:ceph2-0"
2024/08/29 15:41:28 DEBUG : Resolving service "s3" region "HN"
2024/08/29 15:41:28 DEBUG : w0test-0: Need to transfer - File not found at Destination
2024/08/29 15:41:38 DEBUG : w1test-91: md5 = 972ae347f929be9c431a55a8680fa913 OK
2024/08/29 15:41:38 INFO  : w1test-91: Copied (new)
2024/08/29 15:41:38 DEBUG : w1test-86: md5 = 400f7aba4cf2fc6732246cfe37611224 OK
2024/08/29 15:41:38 INFO  : w1test-86: Copied (new)
2024/08/29 15:41:38 DEBUG : w1test-96: md5 = 397ddeb5d3604e528c5c18470fcc9318 OK
2024/08/29 15:41:38 INFO  : w1test-96: Copied (new)
2024/08/29 15:41:38 DEBUG : Waiting for deletions to finish
Transferred:          168 MiB / 168 MiB, 100%, 18.109 MiB/s, ETA 0s
Transferred:          168 / 168, 100%
Elapsed time:         9.6s
2024/08/29 15:41:38 INFO  :
Transferred:          168 MiB / 168 MiB, 100%, 18.109 MiB/s, ETA 0s
Transferred:          168 / 168, 100%
Elapsed time:         9.6s

2024/08/29 15:41:38 DEBUG : 13 go routines active

The log is quite long so I removed the similar part in the middle

1 Like
2

let's compare the timestamps of just one single file.

please post the output of:

  • rclone lsl ceph:ceph2-0/w0test-0
  • rclone lsl cloudian:ceph2-0/w0test-0
3 
[root@rclone ~]# rclone lsl ceph:ceph1-0/w0test-0
  1048576 2024-08-26 17:32:09.267000000 w0test-0
[root@rclone ~]# rclone lsl cloudian:ceph1-0/w0test-0
  1048576 2024-08-26 17:32:09.267000000 w0test-0

but the sync result is not the same i see on cloudian the result is not the same as on rclone server, cloudian shows the time i synced

4

rclone stores mtime as X-Amz-Meta-Mtime

whatever you use to check cloudian it is showing some other field.

I suggest you check with some other tool like aws cli.

5

correct, that is expected behavior.

for s3 providers such as cloudian, aws, wasabi - the time displayed at their websites is the upload time.

whereas, when rclone copies a file, it saves the modification time as metadata.
when rclone compares files, it used the modification time.
if you want rclone to compare files using upload time, use --use-server-modtime

check the rclone docs - https://rclone.org/s3/#modification-times