How to sync S3 with millions of files at root

Help and Support
2

hello and welcome to the forum,

this should not have issues with out of memory.

https://forum.rclone.org/t/recommendations-for-using-rclone-with-a-minio-10m-files/14472/4

rclone lsf -R source:bucket | sort > source-sorted
rclone lsf -R dest:bucket | sort > dest-sorted
comm -23 source-sorted dest-sorted > to-transfer
comm -12 source-sorted dest-sorted > to-delete
rclone copy --files-from to-transfer --no-traverse source:bucket dest:bucket
rclone delete --files-from to-delete --no-traverse dest:bucket

note: for testing,

  1. add --dry-run to these two commands, rclone copy|delete --dry-run ...
  2. use a debug log --log-level=DEBUG --log-file=/path/to/rclone.log