rclone will be listing the 10,000,000 file directory. This will undoubtedly take some time especially over NFS.
Rclone won't start transferring anything from the directory until it has read it all.
Note also that rclone uses getdents64 whereas libc uses getdents; I've seen kernel bugs in getdents64 because it doesn't get used as often (eg in cifs and a go bug report).
How long does rclone size /path/to/10000000filedirectory take?
@Animosity022 's advice for using a newer rclone is good as you'll get a newer go runtime too.