S3 sync memory overflow (leads to OOM)

[kapitainsky]

@minzi_huang

rclone v1.64.0-DEV

Please do not use old rclone version (especially old beta). It is v1.65.1 now

@minzi_huang

Server information for installing rclone: CPU: 12, memory: 16g

It might be indeed not enough memory to sync 12 million objects (it needs at least 12g just to list them). Or just on the edge. This is not something you can change easily. Get more RAM or use below workaround.

If this is one off sync you can resort to this method:

Big syncs with millions of files

Following a RAM problem in the context of several tens of millions of objects, I'd like to share a workaround with you. The problem Rclone syncs on a directory by directory basis. If you have 10,000,000 directories with 1,000 files in and it will sync fine, but if you have a directory with 100,000,000 files in you will a lot of RAM to process it. Until the OOM killer kills the process, the log is then filled by, : 2023/07/06 15:30:35 INFO : Transferred: 0 B / 0 B, -, 0 B/s, ETA …