Rclone mount S3 bucket runs out of memory after many directory listings

[Animosity022]

Check out:

Unable to migrate huge S3 Bucket (500 millions objects / around 100Tb)

Rclone will use something like 1k or RAM per object or folder in a directory, so those 500 million directories will take something like 500 GB of RAM.... What you can do is do a sync using a file list, so something like rclone lsf --files-only -R remote_src:bucket_source > source-files This should complete without using too much RAM. Then you can use this to do the copy. Note the --no-traverse prevents the directory listing and the --no-check-dest prevents rclone checking the source already…

@ncw explains the memory usage pretty well there.