File filtering problem

Help and Support
1

I’m trying to find all RCLONE_TEST files in the tree that are not beneath */RCloneSync/Test/.

The first command demonstrates that --exclude excludes all files from the directory, but when I then add the --include rclone is finding matches in the excluded directory. What is the correct way to do this?


[xxx RCloneSync]$ rclone lsl ..  --exclude "RCloneSync/Test/" | grep TEST
      110 2018-03-14 21:34:18.367576724 RCloneSync/RCLONE_TEST
[xxx RCloneSync]$ rclone lsl ..  --exclude "RCloneSync/Test/" --include RCLONE_TEST
      110 2018-03-14 21:34:18.367576724 RCloneSync/RCLONE_TEST
      110 2018-03-14 21:34:18.000000000 RCloneSync/Test/tests/test_remote_newer - Copy/initial/RCLONE_TEST
      110 2018-03-14 21:34:18.367576700 RCloneSync/Test/tests/test_remote_newer/initial/RCLONE_TEST
      110 2018-03-14 21:34:18.367576700 RCloneSync/Test/tests/test_all/initial/RCLONE_TEST

[xxx RCloneSync]$ rclone -V
rclone v1.37-157-g5a3a56ab\u03b2
- os/arch: linux/amd64
- go version: go1.9
2

My read of https://rclone.org/filtering/#how-the-rules-are-used

Rclone maintains a combined list of include rules and exclude rules.

Each file is matched in order, starting from the top, against the rule in the list until it finds a match. The file is then included or excluded according to the rule type.

I read this as the exclude would be the first match for the files beneath the excluded directory.

3

Note the important section in the docs

You should not use --include* together with --exclude*. It may produce different results than you expected. In that case try to use: --filter*.

So rework your example into

rclone lsl ..  --filter "- RCloneSync/Test/**" --filter "+ RCLONE_TEST" --filter "- *"

And it will hopefully do what you want!

I suggest you try a newer rclone, say 1.42 or the latest beta too :slight_smile:

It should also have printed this warning

$ rclone -v ls --include x --exclude x /tmp/test
2018/06/22 22:16:39 INFO  : Using --filter is recommended instead of both --include and --exclude as the order they are parsed in is indeterminate

But I see the log level needs to be raised so it appears without -v so I’ve done that.

4

Thanks Nick,
Your recommendation is where I ended up.

Note that one cannot predictable use both --filter-from and --filter. With --dump filters I’m seeing that (something like) “+ /*” always ends up at the top of the directory filter list, effectively blocking the <- directories> on the command line.

Luckily, for my application, I don’t need the --filter-from filtered items while at the same time doing the command line. If both --filter-from and command line --filter should play nice together then shall I file a bug?

In any case, an update to the Filtering docs page would be quite helpful, noting not to use both command line --include and --exclude together. I think the current doc says it should work.

FYI, I dropped in 1.42 beta and rclonesync still works fine. I’m adding test case support and need to ignore possible RCLONE_TEST file mismatches in the test directories without tripping the access check fail. I’ll push an update to https://github.com/cjnaz/RCloneSync within a couple days.

5

If you just use one -filter and one --filter-from then things will be consistent, but if you use multiple then you run into the same problem of having multiple --include and --exclude statements.

I think that is probably covered in the docs: https://rclone.org/filtering/#repeating-options both the order that things are processed in and the problems with multiple options.

Nice :smiley: