In case we have a large data set ( millions of files) do we get any performance benefit by using --file-from vs not using this option?
As compared to a non-file, in-line filter? Then no. I'm quite sure there is no difference there aside from the filter file accommodating more complex and nuanced rules.
As compared to no filter, then I suppose it might technically save you some remote listings in some very niche situations, but trying to optimize performance via filtering is likely a red herring. It might be possible sometimes - but only because you aren't using the right tool for the job to begin with.
If you show me your config file (remember to remove sensitive info and password) so I know your cloud service, config (and mount command if any) then I can probably make a suggestion that would be more relevant for actual performance. Describing more about your use-case would help too (size of files, hoe complex is the directory-tree and what do you use these millions of files for?)
This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.