Ok that explains it!
In order to do a sync between two directories rclone currently loads them both into RAM as it can't trust the order of items that the backend delivers them.
Rclone might use ~1k of RAM per object and given that you have a source and a destination 30G of memory doesn't sound completely out of order!
Remember also that since you've set --checkers=64
rclone can be doing up to 64 of these super large directories at once. If you try lowering that number, rclone will use less memory.
This particular problem is talked about in various issues, for example: New listing interface for backends `ListP` paged listing · Issue #4788 · rclone/rclone · GitHub - I'd like to add this to rclone at some point which would fix your problem - maybe you'd like to sponsor the feature?
There are several practical things that can be done to improve things in the mean time. One is to try setting GOGC=20
as outlined here. That trades CPU time for memory and will make a difference.
Lowering --checkers
will help if there is more than one directory with a huge number of files.
Its also possible to use rclone plus some unix tools, for example (choose a separator which doesn't appear in the files - it can be multiple characters). This lists the path, md5 sum and size of every file.
rclone lsf -R --hash MD5 --format phs --files-only --separator "|" src: | sort > src-files
rclone lsf -R --hash MD5 --format phs --files-only --separator "|" dst: | sort > dst-files
Then you can do this to discover the files which need transferring to the dst and deleting from the dst
comm -23 src-files dst-files | cut -d "|" -f 1 > need-transfer
comm -13 src-files dst-files | cut -d "|" -f 1 > need-delete
You can then feed the need-transfer
into rclone copy
and the need delete into rclone delete
.
Or maybe just find the changed files
comm -3 src-files dst-files | cut -d "|" -f 1 | uniq > changed
Then feeding changed into rclone sync --files-from changed src: dst:
might be the best way.