Sync between s3 7billion objects

[asdffdsa (jojothehumanmonkey)]

welcome to the forum,

@allnelone

Please suggest what options I could try adding to speed up the process.

rent a cheap vm in the same region as the bucket.
run rclone on that.

Recommendations for using rclone with a minio 10M+ files

What is the problem you are having with rclone? First and foremost, thanks for the excellent tool, it's fantastic! We have got an in-house data centre serving all our applications and data services (everything on Kubernetes), we rely on kafka-connect-s3 for exporting our Kafka cluster topics to the Minio object-store as a cold backup option for Kafka (similar to https://jobs.zalando.com/en/tech/blog/backing-up-kafka-zookeeper/). The Kafka cluster is pretty big and some of the topics have got 10…

Rclone sync S3 to S3 runs for hours and copy nothing

This is rclone doing HEAD requests to read the modtime most likely. You can stop it doing this with the --size-only or the --checksum flags and the sync should start much quicker. Are a great number of those 80 million files in the same directory? That is what your out of memory makes me think. The problem is big syncs with millions of files in one directory. Rclone syncs on a directory by directory basis so you can have 10,000,000 directories with 1,000 files in and it will sync fine, but …