Help with the "Path Wildcard" filter pattern

[rcloneforum.handsaw5]

What is the problem you are having with rclone?

I'm having trouble understanding the "Filtering, includes and excludes" documentation on the "Path Wildcard" filter pattern. The documentation suggests that the filter pattern dir/** matches folders with the name dir in the remote's root directory, but also folders with the name dir (and their contents) in the remote's subdirectories:

Matches
/dir/anyfile
/subdir/dir/subsubdir/anyfile

Unfortunately, I can't replicate this behaviour using dir/**. To match /subdir/dir/subsubdir/anyfile, I need to run:

rclone ls rclone\ test/ --include '*dir/**'

If the folder that dir is a subdirectory in doesn't have dir in its name, then I need to run:

rclone ls rclone\ test/ --include '**dir/**'

Here's the output of rclone tree for my test folder:

/
├── dir
│   └── anyfile
├── subdir
│   └── dir
│       └── subsubdir
│           └── anyfile
└── subfol
    └── dir
        └── subsubfol
            └── anyfile

7 directories, 3 files

Run the command 'rclone version' and share the full output of the command.

rclone v1.68.2

• os/version: darwin 15.1 (64 bit)
• os/kernel: 24.1.0 (arm64)
• os/type: darwin
• os/arch: arm64 (ARMv8 compatible)
• go/version: go1.23.3
• go/linking: dynamic
• go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Local Filesystem (also tested with Proton Drive)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Local Filesystem

rclone ls rclone\ test/ --include 'dir/**'

Proton Drive

rclone ls Proton\ Drive:rclone\ test/ --include 'dir/**'

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[Proton Drive]
type = protondrive
username = XXX
password = XXX
client_uid = XXX
client_access_token = XXX
client_refresh_token = XXX
client_salted_key_pass = XXX

Due to privacy concerns, I've replaced the username with XXX. The output is otherwise identical.

A log from the command that you were trying to run with the -vv flag

Local Filesystem

2025/01/09 10:12:58 DEBUG : rclone: Version "v1.68.2" starting with parameters ["rclone" "ls" "rclone test/" "--include" "dir/**" "-vv"]
2025/01/09 10:12:58 DEBUG : Creating backend with remote "rclone test/"
2025/01/09 10:12:58 DEBUG : Using config file from "/Users/XXX/.config/rclone/rclone.conf"
2025/01/09 10:12:58 DEBUG : fs cache: renaming cache item "rclone test/" to be canonical "/Users/XXX/Documents/rclone test"
2025/01/09 10:12:58 DEBUG : subdir: Excluded
2025/01/09 10:12:58 DEBUG : subfol: Excluded
2025/01/09 10:12:58 DEBUG : 5 go routines active

Proton Drive

2025/01/09 10:21:37 DEBUG : rclone: Version "v1.68.2" starting with parameters ["rclone" "ls" "Proton Drive:rclone test/" "--include" "dir/**" "-vv"]
2025/01/09 10:21:37 DEBUG : Creating backend with remote "Proton Drive:rclone test/"
2025/01/09 10:21:37 DEBUG : Using config file from "/Users/XXX/.config/rclone/rclone.conf"
2025/01/09 10:21:37 DEBUG : proton drive root link ID 'rclone test': Has cached credentials
2025/01/09 10:21:41 DEBUG : proton drive root link ID 'rclone test': Used cached credential to initialize the ProtonDrive API
2025/01/09 10:21:43 DEBUG : fs cache: renaming cache item "Proton Drive:rclone test/" to be canonical "Proton Drive:rclone test"
2025/01/09 10:21:44 DEBUG : subdir: Excluded
2025/01/09 10:21:44 DEBUG : subfol: Excluded
        0 dir/anyfile
2025/01/09 10:21:46 DEBUG : 6 go routines active

Thank you.

[kapitainsky]

@rcloneforum.handsaw5

dir/**

Use --dump filters flag to have ultimate understanding how rclone interprets your filters:

$ rclone ls . --include "dir1/**" --dump filters
2025/01/09 07:05:37 NOTICE: Automatically setting -vv as --dump is enabled
--- start filters ---
--- File filter rules ---
+ (^|/)dir1/.*$
- ^.*$
--- Directory filter rules ---
+ (^|/)dir1/.*/$
+ (^|/)dir1/$
+ (^|/)dir1/.*$
- ^.*$
--- end filters ---


$ rclone ls . --include "**dir1/**" --dump filters
2025/01/09 07:15:16 NOTICE: Automatically setting -vv as --dump is enabled
--- start filters ---
--- File filter rules ---
+ (^|/).*dir1/.*$
- ^.*$
--- Directory filter rules ---
+ (^|/).*dir1/.*/$
+ (^|/).*dir1/$
+ (^|/).*/$
+ (^|/).*dir1/.*$
- ^.*$
--- end filters ---

And I agree that documentation on filtering subject might be in places not 100% clear (some errors are also possible). Any clarifications/fixes via PR are welcomed.

[rcloneforum.handsaw5]

Thank you for your response, I wasn't aware of the --dump filters flag. I'm having some difficulty parsing the regex filter rules, so just wondering if regex101.com with the Golang flavor selected is a good way to start understanding what these expressions are doing? I'd appreciate your insight, especially if that's not what you would do. Thanks again.

[kapitainsky]

I use regex101 too. What exactly is the issue you are having?

I think with rclone important is to remember that:

{{regexp}}  becomes (^|/)(regexp)$
/{{regexp}} becomes ^(regexp)$

[rcloneforum.handsaw5]

What exactly is the issue you are having?

Just to clarify, I've never worked with regular expressions, so I wanted to see if you thought regex101.com was a good tool for understanding the expressions rclone works with. Since you use it yourself, that's my answer. Thanks for your help, my problem is solved.