Why do we ignore other filtering rules when using the --files-from flag?

What is the problem you are having with rclone?

Other filtering rules are ignored when using --files-from

What is your rclone version (output from rclone version)

v1.49.3

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Ubuntu 18, 64 bits

Which cloud storage system are you using? (eg Google Drive)

Google Cloud Storage

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy <src_path> <dest_path> --files-from --exclude-from

That is in the docs: https://rclone.org/filtering/#files-from-read-list-of-source-file-names

Do you think it should be more obvious? Create an error if attempted or be supported?

I read it in the docs. I am trying to understand why it is not supported. Is there some technical limitation underneath or it's just a decision to not support it? There are valid use cases where both are needed to be used together.

It should definitely throw a critical error to prevent confusion. On a sync you could easily end up deleting files accidentally because you were not aware of this minor detail. It should not accept both parameters if they can't work together in this case.

But I am also curious if there is a technical reason why they can't both be allowed. It seems like that would be useful and the most flexible solution.

FWIW, I've written and am writing a series of simple utilities that have similar flags. I also only allow one or the other. There's no technical limitation per-se. But here's my reasoning:

  • Non-trivial added complexity to include/excluded-related code, and resulting significantly increased opportunity for unexpected results in terms of file selection. Which, as an added bonus, depending on the operation, could be catastrophic relative to user expectations.
  • Allowing multiple methods of include/exclude, greatly increases output complexity (in terms of users' thinking through what the final results of selected files might look like), disproportionate to a slight increase in input complexity (i.e. number and complexity of flags themselves). Surely some kind of exponential function.
    • I say this as the primary user of my scripts/programs. I'm less worried about what other users think, who I might guess are "smarter" or "dumber" than me. I just don't want to screw my files up.
  • If using a --files-from flag, you've presumably generated it using utilities or even just a sequence of grep -iP filters that are ultimately more advanced than what filter flags can offer...e.g. by human inspection, intervention, and intelligence.
  • Or if you'd rather use more complex rclone filters than filter flags allow, you can use --filter-from.

But at minimum for an open source project of volunteers, the first two points -- or even just the first one -- "should" be reason enough to justify it. At least it would be if I were the primary maintainer.

I know the question then usually comes down to, "well how hard could it be"? The answer is invariably, way harder--or more specifically time-consuming--than you'd think. So a perfectly appropriate response to such a hypothetical question (which I don't know if you specifically might be inclined to ask...just generally speaking), would be "why don't you write it up and submit a PR". (To which a perfectly appropriate response to such a hypothetical answer might be, "I don't know how to code", or "My plate is full already, potentially with other open source projects, and I'm looking to be a consumer not a producer of a file sync utility." :grinning:)

Yes the complexity is the reason rclone doesn't do it and generally if someone has gone to the trouble of making a list of files that they want then they don't need them filtered.

I think throwing the error would be the best choice for the time being. Does anyone fancy making an issue for me?

Considering this is just a simple reminder for something we know about, I can probably do that.
I need to put in the 2 we discussed yesterday anyway, so I will just do all 3 then.

1 Like

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.