Question about rule order in --filter-from

[Animosity022]

My explanation did contain a concept of a backup set.

I was trying to say the 'backup set' concept doesn't apply to this discussion in terms of how filters work and the filter concept and how they apply is universal if you are doing a copy / sync / ls / etc command. Apologies if I was confusing my choice of words.

@ncw did respond affirmatively to the following sentence of mine. 'You have neither affirmed nor denied whether the set of files to be operated upon starts as all files in the destination path' in the affirmative. For, he said (with admittedly some unclarity): 'Yes initially all files are passed to the filter'.

I thought I answered this very directly with an affirmative answer. If you type a rclone command, it against the destination. They are caveats to that in terms of how far it goes down the destination based on other filters/flags that you can apply so the answer is quite as direct as yes / no as it depends. Based on a specific question, an affirmative answer can be made.

I am sorry to be increasingly direct but I feel entitled to decent answers to my reasonable questions.

The goal is to help out and give an answer and some of that process is understanding the question as it's not always easy without asking a few questions or having an exchange.

Also, and frankly: I did not in fact in this instance ask you.

I try to help / volunteer my time so @ncw can develop code and if there are questions that are in my technical expertise to answer, I try to do so. I've been wrong before. I'm batting pretty well and will always admit if I make a mistake. That's how we all learn from each other as the community is what makes rclone great.

If we had to depend on ncw to ask everything, he'd never have time to code

[ncw (Nick Craig-Wood)]

Thank you. So, the logic/algorithm is as follows?

Let filter-file be the filter file (the name of which is passed to rclone).
Let rules be the set of rules in filter-file .
Let rule be a (any) single rule within rules .
Let destination be the destination path passed to rclone.
Let item be some (any) file or directory. EDIT: and if the item is a directory then it should be expressed in any rule as item/** .
Let backup be the set of files to be backed-up.

Set backup to contain each item in destination .
For each item within destination
(
For each rule , moving serially through rules
(
If rule matches the full path of item then
(
if rule is an include rule then add item to backup (unless it is in there already).
if rule is an exclude rule then remove item from backup (if it was in there).
Proceed to the next item .
)
)

I think that is pretty much correct. Note that you can have multiple --filter-from and --filter flags which add to the rule set.

Strictly speaking what you've written above is correct for files. For directories the decision rclone makes is whether to recurse into them or not so that truncates the listing. The effect is the same as above though.

[ncw (Nick Craig-Wood)]

I try to help / volunteer my time so @ncw can develop code and if there are questions that are in my technical expertise to answer, I try to do so. I've been wrong before. I'm batting pretty well and will always admit if I make a mistake. That's how we all learn from each other as the community is what makes rclone great.

If we had to depend on ncw to ask everything, he'd never have time to code

And I really appreciate your time on the forum @Animosity022. I try to look at the forums at least once a day, but I also try to do coding too, run the project, make releases etc which means that having help is fantastic

[Animosity022]

Strictly speaking what you've written above is correct for files . For directories the decision rclone makes is whether to recurse into them or not so that truncates the listing. The effect is the same as above though.

I tend to think of it as the rules operate top down against the destination that goes from left to right.

So if you have something like:

test/somefile.txt
blah/somefile.txt
test.txt
hosts

You can grab text files not in the root directory by:

[felix@gemini blah]$ cat ../filtertest
+ **/*.txt
# exclude everything else
- *
[felix@gemini blah]$ rclone ls /home/felix/blah --filter-from ../filtertest
        0 test/somefile.txt
        0 blah/somefile.txt

You can grab all txt files from root onward:

[felix@gemini blah]$ cat ../filtertest
+ *.txt
# exclude everything else
- *
[felix@gemini blah]$ rclone ls /home/felix/blah --filter-from ../filtertest
        0 test.txt
        0 test/somefile.txt
        0 blah/somefile.txt
[felix@gemini blah]$

and so on and so forth so the filters have great impact on how they are created and the order of operation. Knowing the flow of the logic is key as you don't want to match 'too greedy'.

@LinuxOnTheDesktop - can you suggest some changes based on what's written up from what has been shared and I can do a pull request to help clarify things?

[ncw (Nick Craig-Wood)]

Strictly speaking what you've written above is correct for files . For directories the decision rclone makes is whether to recurse into them or not so that truncates the listing. The effect is the same as above though.

I tend to think of it as the rules operate top down against the destination that goes from left to right.

The detail about directories is an optimization thing not a correctness thing (to use computer-science speak) so needn't be part of any overall explanation.

Once tip I can't emphasize enough is use rclone ls to test your filters to see what they do!