Swift sync --checksum calls HEAD on every object so is very slow

Ok that explains it!

In order to do a sync between two directories rclone currently loads them both into RAM as it can't trust the order of items that the backend delivers them.

Rclone might use ~1k of RAM per object and given that you have a source and a destination 30G of memory doesn't sound completely out of order!

Remember also that since you've set --checkers=64 rclone can be doing up to 64 of these super large directories at once. If you try lowering that number, rclone will use less memory.

This particular problem is talked about in various issues, for example: New listing interface for backends `ListP` paged listing · Issue #4788 · rclone/rclone · GitHub - I'd like to add this to rclone at some point which would fix your problem - maybe you'd like to sponsor the feature?

There are several practical things that can be done to improve things in the mean time. One is to try setting GOGC=20 as outlined here. That trades CPU time for memory and will make a difference.

Lowering --checkers will help if there is more than one directory with a huge number of files.

Its also possible to use rclone plus some unix tools, for example (choose a separator which doesn't appear in the files - it can be multiple characters). This lists the path, md5 sum and size of every file.

rclone lsf -R --hash MD5 --format phs --files-only --separator "|"  src: | sort > src-files
rclone lsf -R --hash MD5 --format phs --files-only --separator "|" dst: | sort > dst-files

Then you can do this to discover the files which need transferring to the dst and deleting from the dst

comm -23 src-files dst-files | cut -d "|" -f 1 > need-transfer
comm -13 src-files dst-files | cut -d "|" -f 1  > need-delete

You can then feed the need-transfer into rclone copy and the need delete into rclone delete.

Or maybe just find the changed files

comm -3 src-files dst-files | cut -d "|" -f 1 | uniq > changed

Then feeding changed into rclone sync --files-from changed src: dst: might be the best way.