Rclone Filter file problems when directory included

Help and Support
1

What is the problem you are having with rclone?

Hi - I can't quite seem to get my filter file right. While I figured out this portion, I'm now trying to exclude certain folders in /appdata/ with a different rclone copy command (examples below). The filter file works when I don't explicitly copy /appdata/, but breaks when I include it. I tried to account for this by including **/ BEFORE each folder in filter file, but that didn't solve it.

Run the command 'rclone version' and share the full output of the command.

rclone v1.69.0

  • os/version: slackware 15.0+ (64 bit)
  • os/kernel: 6.6.68-Unraid (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.23.4
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Here is my filter file:

filter file
# Since default is to include all folders in /mnt/user/, exclude main data and unraid folders
- /data/**
- /domains/**
- /isos/**
- /system/**

# Include only backup folders, exclude remaining temporary and replaceable files
- **/plex/Library/Application Support/Plex Media Server/Cache/**
- **/plex/Library/Application Support/Plex Media Server/Media/**
- **/plex/Library/Application Support/Plex Media Server/Metadata/**
- **/plex/Library/Application Support/Plex Media Server/Logs/**
- **/plex/Library/Application Support/Plex Media Server/Crash Reports/**
- **/other/**
- **/jackett/**
- **/QDirStat/**
+ **/radarr/Backups/**
- **/radarr/**
+ **/sonarr/Backups/**
- **/sonarr/**
+ **/lidarr/Backups/**
- **/lidarr/**

# Exclude specific hidden and temporary files
- *fuse_hidden*
- *_HIDDEN
- .recycle**
- .Recycle.Bin/**
- *.backup~*
- *.partial~*

This command works and excludes what I need:

rclone copy "/mnt/user" "gd:backup" "--backup-dir" "gd:backup-deleted" 
        --dry-run \
        --filter-from /mnt/user/appdata/other/rclone/remotes/gd/rclone_filters.txt \

2025/02/15 16:25:57 DEBUG : data: Excluded
2025/02/15 16:25:57 DEBUG : isos: Excluded
2025/02/15 16:25:57 DEBUG : system: Excluded
2025/02/15 16:25:57 DEBUG : domains: Excluded
2025/02/15 16:25:57 DEBUG : appdata: Directory modification time the same (differ by -505.105µs, within tolerance 1ms)
2025/02/15 16:25:57 DEBUG : Added delayed dir = "appdata", newDst=appdata
2025/02/15 16:25:57 DEBUG : appdata/other: Excluded
2025/02/15 16:25:57 DEBUG : appdata/jackett: Excluded
2025/02/15 16:25:57 DEBUG : appdata/QDirStat: Excluded
2025/02/15 16:25:58 DEBUG : appdata/jackett: Excluded
2025/02/15 16:25:58 DEBUG : appdata/QDirStat: Excluded
2025/02/15 16:26:06 DEBUG : appdata/plex/Library/Application Support/Plex Media Server/Crash Reports: Excluded
2025/02/15 16:26:06 DEBUG : appdata/plex/Library/Application Support/Plex Media Server/Logs: Excluded
2025/02/15 16:26:06 DEBUG : appdata/plex/Library/Application Support/Plex Media Server/Cache: Excluded
2025/02/15 16:26:06 DEBUG : appdata/plex/Library/Application Support/Plex Media Server/Media: Excluded
2025/02/15 16:26:06 DEBUG : appdata/plex/Library/Application Support/Plex Media Server/Metadata: Excluded

However, this command then breaks the exclusions with the same filter file. Oddly the only folder that is excluded for some reasons is QDirStat/xdg/config/QDirStat?

rclone copy "/mnt/user/appdata" "gd:backup" "--backup-dir" "gd:backup-deleted" 
        --dry-run \
        --filter-from /mnt/user/appdata/other/rclone/remotes/gd/rclone_filters.txt \

2025/02/15 16:41:01 DEBUG : QDirStat/xdg/config/QDirStat: Excluded

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[gd]
type = drive
client_id = XXX
client_secret = XXX
scope = drive
token = XXX
root_folder_id = XXX
server_side_across_configs = true
team_drive =
2 
**/something

is not the same as

**something

$ tree .
.
└── directory

# this is your second example not working. There is no match for this filter.`/` ≠ `**/`
$ rclone tree . --filter "- **/directory/**" --dump filters
--- start filters ---
--- File filter rules ---
- (^|/).*/directory/.*$
--- Directory filter rules ---
- (^|/).*/directory/.*$
--- end filters ---
/
└── directory

# `/` matches `**`
$ rclone tree . --filter "- **directory/**"
--- start filters ---
--- File filter rules ---
- (^|/).*directory/.*$
--- Directory filter rules ---
- (^|/).*directory/.*$
--- end filters ---
/

# or anchor filter at root explicitly 
$ rclone tree . --filter "- /directory/**"
/

why oddly when you have filter - **/QDirStat/** and there is matching directory

please note that it is not the same directory as in your first example. That it starts with QDirStat is irrelevant but maybe confusing. Filter matches QDirStat sub dir in QDirStat root dir:)

vs.

I would say that it is much safer to anchor your rules to some root explicitly. Otherwise your rules can match some other directories (like with QDirStat example). Decide what your rclone copy command source dir is and create explicit filters.

Simple matching rules have their limits and if you require something more complex, safe and working then start learning regular expressions:)

3

Thanks for the response.

The issue is this filter is part of a complicated script where I have different rclone commands copying different directories. Appdata is separate because I need to do a few separate commands and pause/unpause running applications before backing it up. I'd rather not create two filter files as I'm trying to make it more extensible.

So, I'm trying to have one filter file that will always exclude these directories, no matter what the copy command source directory is :blush:

I didn't fully understand your post. I need to add this regex to filter file in order to get the directory to be excluded no matter where this folder is in the source directory?

# If appdata is included, include only necessary backup folders, and exclude remaining temporary and replaceable directories using a regex

- {{(^|/).*/plex/Library/Application Support/Plex Media Server/Cache/.*$}}
4

What you need is instead of:

+ **/radarr/Backups/**
- **/radarr/**
+ **/sonarr/Backups/**
- **/sonarr/**
+ **/lidarr/Backups/**

use

+ **radarr/Backups/**
- **radarr/**
+ **sonarr/Backups/**
- **sonarr/**
+ **lidarr/Backups/**

to make it work for your both sync commands.

Potential problem is that these patters can match different items than intended. You could see unexpected example in case of "QDirStat". To do this right you have to create more complex filters using regular expressions.

But IMO more complex you make it then more likely you will create bugs and make mistakes. Keep it simple. I think I would rethink my script or use two different filter files with very explicit rules.

5

But if you insist on using one filter list for your two sync commands then what you need is regex which will exclude for example both:

/appdata/jackett/**

and

/jackett/**

For example I think this would do:

- /{{(?:appdata/)?jackett/.*}}

or

- /{{(appdata/jackett|jackett)/.*}}

Remember that:

{{regexp}}  becomes (^|/)(regexp)$
/{{regexp}} becomes ^(regexp)$

You can use https://regex101.com/ to test.

With simple rules:

  1. - **/jackett/** does not work with one of your syncs
  2. - **jackett/** will work with both but will also exclude /some/other/path/jackett/** which can lead to hard to detect errors in your sync
1 Like
6

I see. Thanks! I decided to keep it simple with the simple rules. I'm not worried about case 2 for now, but may switch to regex if I run into some cases there.

7

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.