Why does the --filter flag return results unfiltered?

Help and Support
1

The installed version of rclone is rclone v1.50.2 on Ubuntu 18.04 LTS. The remote is DropBox.

When attempting to run the command rclone ls --filter '+ *.pdf' dropbox:, the results include files with extensions other than .iso. For example, .docx, .txt and so on.

If the command rclone ls --include '*.pdf' dropbox: is used, it returns only files with a pdf file extension whereas rclone ls --include '+ *.pdf' dropbox: results in no files.

In contrast the command rclone ls --filter '*.pdf' dropbox: displays the error Failed to load filters: malformed rule "*.pdf"

2

Check out:

https://rclone.org/filtering/

They have different syntax between the options available.

3

--include puts an implicit exclude all on the end of the list. If you are using --filter you need to add it yourself

rclone ls --filter '+ *.pdf' --filter '- *' dropbox:
1 Like
4

Thanks for clearing that up @ncw. It would suggest that --include is far more efficient and effective. What are some examples --filter could be used for that --include could not or should not?

5

If you read the link above I had posted, it gives use cases too:

https://rclone.org/filtering/

6

I did read it before posting however it isn't clear. It may be the way I understood it.

For example, the excerpt below (emphasis mine) suggests that --filter should be used for a single include or exclude. It suggests (based on my interpretation) that if I construct a command such as --filter '+ *.pdf, it should only return the output with files that have a .pdf extension which isn't the case.

This can be used to add a single include or exclude rule. Include rules start with + and exclude rules start with - . A special rule called ! can be used to clear the existing rules.

This flag can be repeated. See above for the order the flags are processed in.

Eg --filter "- *.bak" to exclude all bak files from the sync.

7

By default, it's inclusive, so you'd have to filter out everything else:

Working:

felix@gemini:~$ rclone ls --filter '+ *.pdf' --filter '- *' /home/felix/test --dump filters
--- start filters ---
--- File filter rules ---
+ (^|/)[^/]*\.pdf$
- (^|/)[^/]*$
--- Directory filter rules ---
+ ^.*$
- ^.*$
--- end filters ---
        0 test.pdf

Not working:

felix@gemini:~$ rclone ls --filter '+ *.pdf' /home/felix/test --dump filters
--- start filters ---
--- File filter rules ---
+ (^|/)[^/]*\.pdf$
--- Directory filter rules ---
+ ^.*$
--- end filters ---
      221 blah
       13 filter
      221 hosts
        0 test.pdf
      221 test1

or you can filter out things with a one line.

felix@gemini:~$ rclone ls --filter '- *.pdf' /home/felix/test --dump filters
--- start filters ---
--- File filter rules ---
- (^|/)[^/]*\.pdf$
--- Directory filter rules ---
--- end filters ---
      221 blah
       13 filter
      221 hosts
      221 test1
8

Thanks @Animosity022. The examples are great. It isn't clear though..

For example why I would use --filter '+ *.pdf --filter '- *' if I can simply use--include '*.pdf. The former seems far more complicated in that I have to explicitly exclude all.

--exclude '*.pdf' and --filter '- *.pdf' seem to result in the same output.

9

I personally would use what works best for the situation. You have options to use what works best for your use case so not everything is black and white.

For your example, I'd just use include as that best and most simply solves the problem.

filter gives much more flexibility in terms of setting up a longer list.

A hammer opens everything but it is not always the right tool to use :slight_smile:

10

:laughing:, yes I am in complete agreement in not using a hammer for everything. I was hoping that you may be able to shed light on examples of when --filter is a better choice than --include. The reason being is that it would help filter (pun intended) the choices in using either flags.

11

Filter would be used if you would have a more complex set of rules that you wanted to process and most folks use filter-from so they have it all in a file.

It does top down processing of the rules and you have a lot of flexibility to create a set of rules that meet some interesting needs.

The example here:

https://rclone.org/filtering/#filter-from-read-filtering-patterns-from-a-file

Gives a pretty good one. Also, you can't mix include and exclude together so in cases where you need both an include and exclude, filter would be the option.

12

Thanks. The inability to mix include and exclude hit in on the nail :smile:

1 Like
13

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.