Pattern for filter-from to exclude directories that have matching files

[phirestalker]

STOP and READ USE THIS TEMPLATE NO EXCEPTIONS - By not using this, you waste your time, our time and really hate puppies. Please remove these two lines and that will confirm you have read them.

What is the problem you are having with rclone?

I want to match files with a certain pattern and if they match exclude the directory under the root that contains them. I have tried multiple versions, but the file movie_filter.txt contains the single line

- [Ss]?[0-9][0-9]?[xXeE.\-][0-9][0-9]?

Run the command 'rclone version' and share the full output of the command.

rclone v1.67.0

• os/version: ubuntu 22.04 (64 bit)
• os/kernel: 5.15.0-117-generic (x86_64)
• os/type: linux
• os/arch: amd64
• go/version: go1.22.4
• go/linking: static
• go/tags: none

Which cloud storage system are you using? (eg Google Drive)

webdav

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount --daemon alldebrid:/magnets /home/phire/docker/jellyfin/alldebrid/movies --umask 0222 --allow-other --filter-from /etc/rclone/movie_filter.txt

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[alldebrid]
type = webdav
url = https://webdav.debrid.it
vendor = other
user = XXX
pass = XXX

A log from the command that you were trying to run with the -vv flag

2024/07/31 08:39:45 DEBUG : Failed to find user cache dir, using temporary directory: neither $XDG_CACHE_HOME nor $HOME are defined
2024/07/31 08:39:45 DEBUG : rclone: Version "v1.67.0" starting with parameters ["rclone" "mount" "-vv" "--daemon" "alldebrid:/magnets" "/home/phire/docker/jellyfin/alldebrid/movies" "--umask" "0222" "--allow-other" "--filter-from" "/etc/rclone/movie_filter.txt"]
2024/07/31 08:39:45 DEBUG : Creating backend with remote "alldebrid:/magnets"
2024/07/31 08:39:45 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2024/07/31 08:39:45 DEBUG : found headers: 
2024/07/31 08:39:46 DEBUG : fs cache: renaming cache item "alldebrid:/magnets" to be canonical "alldebrid:magnets"
2024/07/31 08:39:46 DEBUG : rclone: Version "v1.67.0" finishing with parameters ["/usr/bin/rclone" "mount" "-vv" "--daemon" "alldebrid:/magnets" "/home/phire/docker/jellyfin/alldebrid/movies" "--umask" "0222" "--allow-other" "--filter-from" "/etc/rclone/movie_filter.txt"]

[asdffdsa (jojothehumanmonkey)]

hi,

@phirestalker

I want to match files with a certain pattern and if they match exclude the directory under the root that contains them

might be able to use --exclude-if-present

[phirestalker]

just putting the pattern after --exclude-if-present did not work. Is there something I should add to the pattern?

[asdffdsa (jojothehumanmonkey)]

fwiw, much easier to test using rclone ls, not complex rclone mount

rclone ls --dump=filters --include="[Ss]?[0-9][0-9]?[xXeE.\-][0-9][0-9]?"

@phirestalker

Is there something I should add to the pattern?

post a few examples of the files that you want to match

[phirestalker]

The.Man.From.Uncle.(1964).Complete-Z0DiAC99/The Man From Uncle - 4x15 - The Seven Wonders of The World Affair, Part 1.mp4
The Twilight Zone (1959) Season 1-5 S01-05 (1080p BluRay x265 HEVC 10bit AAC 2.0 ImE)/The Twilight Zone (1959) - S05E28 - Caesar and Me (1080p BluRay x265 ImE).mkv

[asdffdsa (jojothehumanmonkey)]

please, bold the exact test you want to match?

[phirestalker]

edited last post

[asdffdsa (jojothehumanmonkey)]

this is not 100% perfect but the pattern will match both files.
can take that and tweak it for edge cases.
sorry, i do not have the time to create the perfect pattern for you.

rclone ls d:/tmp --include="*{{\dx\d\d|S\d\dE\d\d}}*" 
      125 The Man From Uncle - 4x15 - The Seven Wonders of The World Affair, Part 1.mp4
      125 The Twilight Zone (1959) - S05E28 - Caesar and Me (1080p BluRay x265 ImE).mkv

[kapitainsky]

There is no one way to solve it. You have to think what exactly is pattern your want to match. And as @asdffdsa mentioned there is no perfect one - especially for very vague requirement:)

Based on your example I would go with filter:

+ {{(.*) - S\d{1,}E\d{1,} - (.*)}}
+ {{(.*) - \d{1,}x\d{1,} - (.*)}}
- *

Use https://regex101.com/ for your playground with regex.

Hopefully both our examples give you idea how to approach it.

[phirestalker]

Thanks for the help so far. I was wondering if --exclude-if-present even uses pattern matching this way or only simple exact matching?

[asdffdsa (jojothehumanmonkey)]

@phirestalker

I was wondering if --exclude-if-present even uses pattern matching this way or only simple exact matching?

i did a quick test, and looks like --exclude-if-present does not support patterns.

[kapitainsky]

@asdffdsa

looks like --exclude-if-present does not support patterns

Something new for me... But indeed in docs:

Exclude directory based on a file

The --exclude-if-present flag controls whether a directory is within the scope of an rclone command based on the presence of a named file within it. The flag can be repeated to check for multiple file names, presence of any of them will exclude the directory.

"named file"....

So its purpose is more like for other software respecting CACHEDIR.TAG or .ignore files to exclude directories.

[kapitainsky]

Power of Unix tools to the rescue:) You could:

1. list all files matching your pattern -> save to file list.txt

rclone lsf remote: --format p -R --filter XYZ > list.txt

2. Strip filenames leaving only directories:

awk ' { sub("/[^/]*$", "/") } 1' list.txt > dirs.txt

3. Remove duplicates

sort dirs.txt | uniq -u > unique_dirs.txt

Use this list to build filter using --filter-from for whatever operation you are doing.

[phirestalker]

It looks like I can definitely exlude files, but not the directory that has them when there are other unmatching files in there. I will have to look at writing some kind of tool on top of rclone for this.