I'm trying to optimize the copy of hundreds thousands of small files using the webdav protocol. Source is the webdav server, destination is a local folder. Bandwidth doesn't seem to be a limited factor.
The server seems to be IO limited.
I've tried different options, using --check-first then --checkers=50 but with limited success.
Seems that parsing the source folder to check which files must be copied is so slow that the transfers can't be done during the night. I can check around 20 gb in 60 min, so it will take 30h to check the 600 go, not even talking about transferring the files.
Is there any optimization I could use to reduce this checking time?