Source Address（10.49.1.236） Information: There are 518 files in a small directory, and 10,000,000 files in each of the other eight directories. Small directory transfer is normal, but when transferring eight large directories, it seems to be stuck. The rclone log does not change except for time. (10,000 files were detected when testing 1 million files before, and then started to transfer) It looks like it's going to read the catalog all the time. It won't stop. Ouch.. Tracking threads find that LSTAT and getdent64 are also called every minute, but most of the time they are stuck in futex. By using the grab package, it was found that it was still reading the catalog, but it had been reading for more than ten hours,and the number of files detected should have already exceeded 10,000.
Oh, Reading directories and selecting files are two steps. Max-backlog only works in the selection files stage, but it still needs to read directories first.this takes up memory, so swap guarantees adequate memory.
Does rclone need to read all directories in advance, or is it just a directory of 10 million files?
It took me less than three minutes to execute rclone size directly on the NFS server where I stored 10,000,000 file directories. But I executed rclone size on the NFS client that mounted this directory, and it's not over in 150 minutes.
I'll test the speed of the latest version 1.49.3 next.
NFS does slow down reading directories, but there may be other problems. I executed 'rclone size' in the directory of 100,000 files on the NFS client for 1 minute and 4 seconds. 'rclone size' was executed in the same environment, the directory of one million files was executed for 10 minutes, and the directory of ten million files was executed for nearly a day but still not finished. NFS uses xfs, and NFS client is a virtual machine with 4 GB memory. This may be related to indirect inode or swap, but it's still strange.
I tested it under 100,000 files. 'du -hs' is about 30% slower than 'rclone size', probably because gendents64 performs better.
Indeed, I executed the command 'du -hs' for nearly ten hours, and it was still incomplete.I find that "du -hs" reads 100,000 documents at a time, starting from scratch every time! And the command 'getdents' reads 50 directories at a time, but the cookie of the getdents read directory is not the last cookie of the last getdents directory at a time.These two problems slowed down later.So gendents take up more and more time later.So that should be the reason.
Swap did occur during rclone's transmission of 10 million files, and the last picture on the first floor was the memory scene at that time.