Lsf (and others) are still displaying filtered names

Suspected Bug
1

What is the problem you are having with rclone?

lsf (and probably other listing commands) are outputting all subdirs, even excluded ones. See the command section for more details and tests.

I suspect it is related to allowing them to pass for the sake of walking the tree but they should be filtered out of the final display!

Discourse suggests this post but that seems to be an issue with the filtering syntax. This is pretty simple filters.

What is your rclone version (output from rclone version)

rclone v1.57.0
- os/version: darwin 10.15.7 (64 bit)
- os/kernel: 19.6.0 (x86_64)
- os/type: darwin
- os/arch: amd64
- go/version: go1.17.2
- go/linking: dynamic
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Local to test and demo. Seen on OneDrive and SFTP

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Consider the following files: (as per rclone tree .)

/
├── file1.keep
├── subdirA
│   └── file2.exc
├── subdirB
│   ├── file3.exc
│   └── file4.keep
├── subdirC.keep
│   └── file5.exc
├── subdirD.keep
│   └── file6.keep
├── subdirE.exc
│   └── file7.exc
└── subdirF.exc
    └── file8.keep

And the following outputs: (the |sort is just to ensure the display is the same)

--include Based

$ rclone lsf -R . --include "*.keep"|sort
file1.keep
subdirA/
subdirB/
subdirB/file4.keep
subdirC.keep/
subdirD.keep/
subdirD.keep/file6.keep
subdirE.exc/
subdirF.exc/
subdirF.exc/file8.keep

Equivalent --filter based:

$ rclone lsf -R . --filter "+ *.keep" --filter "- **"|sort
file1.keep
subdirA/
subdirB/
subdirB/file4.keep
subdirC.keep/
subdirD.keep/
subdirD.keep/file6.keep
subdirE.exc/
subdirF.exc/
subdirF.exc/file8.keep

An additional explicit exclude

$ rclone lsf -R . --filter "+ *.keep" --filter "- **" --filter "- *.exc" |sort
file1.keep
subdirA/
subdirB/
subdirB/file4.keep
subdirC.keep/
subdirD.keep/
subdirD.keep/file6.keep
subdirE.exc/
subdirF.exc/
subdirF.exc/file8.keep

And just for good measure, Be even more explicit

$ rclone lsf -R . --filter "+ *.keep" --filter "- **" --filter "- *.exc" --filter "- *.exc/**" |sort
file1.keep
subdirA/
subdirB/
subdirB/file4.keep
subdirC.keep/
subdirD.keep/
subdirD.keep/file6.keep
subdirE.exc/
subdirF.exc/
subdirF.exc/file8.keep

Obviously, I could just use --files-only but that is (a) a workaround to the actual problem and (b) still shows (implicitly and explicityly) excluded directory listings.

The rclone config contents with secrets removed.

N/A (local)

A log from the command with the -vv flag

Just from the initial

2021/12/30 08:00:23 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "lsf" "-R" "." "--include" "*.keep" "-vv"]
2021/12/30 08:00:23 DEBUG : Creating backend with remote "."
2021/12/30 08:00:23 DEBUG : Using config file from "/Users/<REDACTED>/.config/rclone/rclone.conf"
2021/12/30 08:00:23 DEBUG : fs cache: renaming cache item "." to be canonical "/Users/<REDACTED>/test"
2

I agree, the examples you show are confusing (at first), but they seem correct according to the docs (to me):

How filter rules are applied to directories

Rclone commands are applied to path/file names not directories. ...
Directory filter rules are defined with a closing / separator.

Source: https://rclone.org/filtering/#how-filter-rules-are-applied-to-directories

I therefore suggest we live with this as a quirk and focus our limited development resources on more critical issues and the PR's pending in review.

3

First of all,

Totally reasonable! I would do the same. I think I will still open a ticket so it is documented if we agree it is a bug but it is probably not worth the effort to fix. Especially since it may be a complicated fix to add another filtering pass

But anyway

I added another filter:

$ rclone lsf -R . --filter "+ *.keep" --filter "- **" --filter "- *.exc" --filter "- *.exc/**" --filter "- *.exc/" |sort
file1.keep
subdirA/
subdirB/
subdirB/file4.keep
subdirC.keep/
subdirD.keep/
subdirD.keep/file6.keep
subdirE.exc/
subdirF.exc/
subdirF.exc/file8.keep

and it is still there.

4

I am hesitating to call it a bug, because one could argue that it is an accepted oddity to simplify the code (and thereby save resources).

Personally, I wouldn’t create an issue because it will detract focus from other and more important tasks - e.g. developers reading, understanding and prioritizing this issue instead of fixing/reviewing other more critical issues/improvements.

The decision is however fully up to you, just make sure that you add one or more use cases where fixing this will add significant value to rclone.

1 Like
5

did someone summon me?

good point

gooder point

goodest point

1 Like
6

I did end up creating an issue (#5891). To be fair, it was before Ole suggested not to...

I guess this gets philosophical, but to me if a behavior is not expected (e.g. displaying excluded directories) and it is not documented behavior, then it is a bug. And I think all bugs should be noted. That is not the same as saying all bugs should be addressed! The resolution is to document the behavior (however odd), or fix it. And of course, not doing anything is 100% reasonable too.

7

It's good to open a bug so it can be flagged as a known item and depending on how many folks may run across it, it can influence if it gets addressed or not.

Sometimes it just takes one person reporting something to get a flood of others that have the same thing. Who knows. Just report it I'd say as it can always be closed / addressed / marked as known / etc.

1 Like
8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.