Sync between s3 7billion objects

[allnelone]

What is the problem you are having with rclone?

I need to set up synchronization between two object storages in different locations, syncing 7 million objects from one bucket to another, with hash comparison. The synchronization needs to run every 4 hours. I'm struggling to find options that allow the process to complete within 2 hours. I've used transfers and checkers set to 900, as well as the fast-list option. On average, synchronization takes 4 hours, sometimes even longer. During synchronization, less than 10,000 objects are copied at a time, and the volume is under 5GB. Please suggest what options I could try adding to speed up the process.

Run the command 'rclone version' and share the full output of the command.

rclone v1.73.1

• os/version: ubuntu 22.04 (64 bit)
• os/kernel: 5.15.0-60-generic (x86_64)
• os/type: linux
• os/arch: amd64
• go/version: go1.25.7
• go/linking: static
• go/tags: none

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync src:backet dst:backet -vv --checksum --transfers=900 --checkers=900 --fast-list --log-file /logs/$(date -I).rclone.log

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[dst]
type = s3
provider = AWS
access_key_id = XXX
secret_access_key = XXX
endpoint = dst
acl = private

[src]
type = s3
provider = AWS
access_key_id = XXX
secret_access_key = XXX
endpoint = src
acl = authenticated-read
### Double check the config for sensitive info before posting publicly

A log from the command that you were trying to run with the -vv flag

2026/03/04 12:03:32 INFO  :
Transferred:      560.641 MiB / 560.641 MiB, 100%, 405.887 KiB/s, ETA 0s
Errors:                60 (retrying may help)
Checks:          22013856 / 22013856, 100%, Listed 44078563
Transferred:         6142 / 6142, 100%
Elapsed time:   4h52m28.3s

2026/03/04 12:03:32 DEBUG : 657 go routines active

[asdffdsa (jojothehumanmonkey)]

welcome to the forum,

@allnelone

Please suggest what options I could try adding to speed up the process.

rent a cheap vm in the same region as the bucket.
run rclone on that.

Recommendations for using rclone with a minio 10M+ files

What is the problem you are having with rclone? First and foremost, thanks for the excellent tool, it's fantastic! We have got an in-house data centre serving all our applications and data services (everything on Kubernetes), we rely on kafka-connect-s3 for exporting our Kafka cluster topics to the Minio object-store as a cold backup option for Kafka (similar to https://jobs.zalando.com/en/tech/blog/backing-up-kafka-zookeeper/). The Kafka cluster is pretty big and some of the topics have got 10…

Rclone sync S3 to S3 runs for hours and copy nothing

This is rclone doing HEAD requests to read the modtime most likely. You can stop it doing this with the --size-only or the --checksum flags and the sync should start much quicker. Are a great number of those 80 million files in the same directory? That is what your out of memory makes me think. The problem is big syncs with millions of files in one directory. Rclone syncs on a directory by directory basis so you can have 10,000,000 directories with 1,000 files in and it will sync fine, but …

[allnelone]

The virtual machine is already in the same region. I have considered obtaining listings (e.g., using the lsf command), but I thought there might be some options for the sync command itself. The --s3-upload-cutoff option is not suitable because the files are small (less than 1 GB), and the most time-consuming part is building the file list. In one bucket, running lsf takes about 15–20 minutes; in another, it takes around an hour. The remaining time is spent comparing hash sums. I am specifically interested in speeding up the listing and hash comparison stages—the actual copying of new files is fine and does not take much time.

[asdffdsa (jojothehumanmonkey)]

@allnelone

speeding up the listing and hash comparison stages

as for listing, might be able to use --max-age and/or --use-server-modtime
https://rclone.org/s3/#avoiding-head-requests-to-read-the-modification-time

as for hash, if the files in the src bucket and dst bucket were upload by rclone, then rclone does not have to compute checksums.

[allnelone]

I'm not sure if I understood the suggestion correctly. The idea is to check checksums only for new files that have been modified within a certain time period? And also build the file list based on that for the sync command. Suppose we run the command:

rclone sync src:bucket dst:bucket -vv --checksum --transfers=900 --checkers=900 --fast-list --max-age=1d --use-server-modtime

Will it work as follows: it looks at the modification time of the object in src, looks at the modification time of the object in dst, and if the object's modification time is older than 1 day, it skips those objects? And for all others, it applies listing and checksum comparison. Is that correct?

[asdffdsa (jojothehumanmonkey)]

@allnelone

Will it work as follows

do the test, then you will know. use --dry-run and read the debug log

[allnelone]

Thank you for your help. I’m going to try using the implementation with --max-age and --use-server-modtime. Based on my tests, this appears to be the most optimal approach.

[asdffdsa (jojothehumanmonkey)]

@allnelone

Thank you for your help

welcome

---

@allnelone

I’m going to try using the implementation with --max-age and --use-server-modtime. Based on my tests, this appears to be the most optimal approach.

rclone has a lot of flags. and each use-case is different.
good, that you figured out what works best in your case.

[ncw (Nick Craig-Wood)]

@allnelone this sounds like an Enterprise problem so if you would like some support then If you are working on behalf of a company you might be interested in taking out a support contract which can help you get answers quicker and keeps the rclone project sustainable.