Calling RClone with filtering --include vs. multiple calls

Help and Support
1

Hi All,

This is a very niave question, but presumably it is much better when copying only a subset of directories within a parent directory, to call rclone once with --include filtering rather than multiple calls, once per path. I would guess this because RClone can then take advantage of multi-threading and possibly other benefits.

A couple of questions mainly out of interest:

  1. is this correct?
  2. along with multi-processing, are there other benefits to a single-call?
  3. If true, I might suggest adding a top-line to the filtering docs staying the benefit of using this rather than multiple calls, for the niave user like me :slight_smile:

Best Wishes,
Joe

2

Hi Joe,

Sometimes, sometimes not - it heavily depends on your data (number of folders, folderlevels, files per folder, ...)

So it is impossible to give a good answer.

I would start with the simplest in the situation, and then test the alternatives if it seems slow/inefficient.

1 Like
3

Using one (complicated) set of filters has the following advantages

  • Only one backend is made and this may have some overhead as some backends need to do an API call on startup.
  • Only one set of listing commands are done. This may or may not be important depending on exactly what your filters are.

And some disadvantages

  • Filters are complicated and can be hard to maintain
  • Filters interfere with --fast-list so if you are using filters and --fast-list rclone will list the entire tree before filtering.

So as Ole said, measure first!

1 Like
4

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.