Struggling with --filter-from copying unwanted subdirs 😩

chelming · May 10, 2024, 2:46am

What is the problem you are having with rclone?

I'm an idiot and don't understand filters. I'm attempting to use a minimal filter based on the docs, but from what I can tell it's not matching as I'd expect. I have a bunch of folders/files I'd like to sync but it keeps copying everything. I've pared my filter file down to two lines, an include for a specific file and then an exclude for everything else to try and start small and get that working but it seems like it's still trying to copy things in subdirectories.

Eventually I want to get to the point where I copy appdata/homeassistant/config/backups/* but not the rest of the home assistant directory. do I need to - each part of the path before getting there? Something like

- appdata/*
- appdata/homeassistant/*
- appdata/homeassistant/config/*
+ appdata/homeassistant/config/backups/*

Regardless of that, I can't even seem to get "copy a single file and ignore everything else" working as I'd expect.

Run the command 'rclone version' and share the full output of the command.

rclone v1.66.0

os/version: ubuntu 22.04 (64 bit)
os/kernel: 5.15.0-105-generic (x86_64)
os/type: linux
os/arch: amd64
go/version: go1.22.1
go/linking: static
go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone sync /media/docker "drive:Backup/Docker" --filter-from /media/docker/slim.txt -i --dump filters

my slim.txt file contains the following (I've also tried - **):

+ rclone-filter.txt
- *

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[drive]
type = drive
client_id = XXX
client_secret = XXX
scope = drive
team_drive =
token = XXX

A log from the command that you were trying to run with the `-vv` flag

2024/05/09 22:35:08 NOTICE: Automatically setting -vv as --dump is enabled
--- start filters ---
--- File filter rules ---
+ (^|/)rclone-filter\.txt$
- (^|/)[^/]*$
--- Directory filter rules ---
+ ^.*$
- ^.*$
--- end filters ---
2024/05/09 22:35:08 DEBUG : rclone: Version "v1.66.0" starting with parameters ["rclone" "sync" "/media/docker" "drive:Backup/Docker" "--filter-from" "/media/docker/slim.txt" "-i" "--dump" "filters"]
2024/05/09 22:35:08 DEBUG : Creating backend with remote "/media/docker"
2024/05/09 22:35:08 DEBUG : Using config file from "/home/chris/.config/rclone/rclone.conf"
2024/05/09 22:35:08 DEBUG : Creating backend with remote "drive:Backup/Docker"
2024/05/09 22:35:08 DEBUG : Google drive root 'Backup/Docker': 'root_folder_id = 0AAxeANHaf86DUk9PVA' - save this in the config to speed up startup
2024/05/09 22:35:09 DEBUG : docker-compose.yml: Excluded (Path Filter)
2024/05/09 22:35:09 DEBUG : docker-compose.yml: Excluded
2024/05/09 22:35:09 DEBUG : appdata: Directory modification time the same (differ by -155.921µs, within tolerance 1ms)
2024/05/09 22:35:09 DEBUG : compose: Directory modification time the same (differ by -407.567µs, within tolerance 1ms)
2024/05/09 22:35:09 DEBUG : secrets: Directory modification time the same (differ by -93.745µs, within tolerance 1ms)
2024/05/09 22:35:09 DEBUG : rclone-filter.txt: Size and modification time the same (differ by -391.187µs, within tolerance 1ms)
2024/05/09 22:35:09 DEBUG : rclone-filter.txt: Unchanged skipping

rclone: make directory "appdata/audiobookshelf"?
y) Yes, this is OK (default)
n) No, skip this
s) Skip all make directory operations with no more questions
!) Do all make directory operations with no more questions
q) Exit rclone now.
y/n/s/!/q> 2024/05/09 22:35:09 DEBUG : compose/snippet-box.yml: Excluded (Path Filter)

The part that confuses me is: rclone: make directory "appdata/audiobookshelf"

Why is it asking me to create that directory if the only include is rclone-filter.txt and everything else should be excluded? Is it because the appdata directory exists on the remote?

kapitainsky · May 10, 2024, 4:57am

So assuming that this directory has /media/docker/appdata...path, you want something like:

+ /appdata/homeassistant/config/backups/**
- *

Full path is synced not just one file, so if your rclone-filter.txt path contains appdata/audiobookshelf then these directories have to be created.

jwink3101 · May 10, 2024, 7:30am

You got the answer (I think) but let me explain why this is happening.

As soon as a filter rule is matched the file is either added or skipped based on the "+" or -".

So when you have

- appdata/*
- appdata/homeassistant/*
- appdata/homeassistant/config/*
+ appdata/homeassistant/config/backups/*

you will never get to your last line. Side note, you probably also want ** to match subdirs. So try:

+ appdata/homeassistant/config/backups/*
- appdata/*
- appdata/homeassistant/*
- appdata/homeassistant/config/*

(or the same with **).

But actually, you don't even need all of this. Just

+ appdata/homeassistant/config/backups/*
- appdata/**

will work.

One downside to this though is that rclone can't be smart about not listing appdata since it won't know if it has a matching pattern. There is no way around this if you have other stuff to sync but if not, you're better off with

rclone sync source:appdata/homeassistant/config/backups/ dest:appdata/homeassistant/config/backups/

rather than filters.

Additional note:

This is why it is not a good idea to mix --filter with --include and --exclude (and related). While the order that rclone considers them is documented, (a) it is error prone and (b) you can't mix and match + and - for better contro.

chelming · May 10, 2024, 11:57am

this is the part that's confusing me.

/media/docker
├── appdata
│   ├── audiobookshelf
│   ├── other directories
├── compose
│   ├── audiobookshelf.yml
│   ├── other yammels
├── secrets
│   ├── some env files
├── docker-compose.yml
├── rclone-filter.txt
└── slim.txt

$ sudo find /media/docker -name 'rclone-filter.txt'
/media/docker/rclone-filter.txt

rclone-filter.txt is only in the root of the directory I'm trying to copy.

here's what I just tried (going with the belt and suspenders approach)

$ cat /media/docker/slim.txt
+ rclone-filter.txt
- appdata
- appdata/
- appdata/*
- appdata/**
- *
- **

$ rclone sync /media/docker "drive:Backup/Docker" --filter-from /media/docker/slim.txt -i --dump filters
2024/05/10 08:00:19 NOTICE: Automatically setting -vv as --dump is enabled
--- start filters ---
--- File filter rules ---
+ (^|/)rclone-filter\.txt$
- (^|/)appdata$
- (^|/)appdata/.*$
- (^|/)appdata/[^/]*$
- (^|/)[^/]*$
- (^|/).*$
--- Directory filter rules ---
+ ^.*$
- (^|/)appdata/.*$
- ^.*$
- (^|/).*$
--- end filters ---
2024/05/10 08:00:19 DEBUG : rclone: Version "v1.66.0" starting with parameters ["rclone" "sync" "/media/docker" "drive:Backup/Docker" "--filter-from" "/media/docker/slim.txt" "-i" "--dump" "filters"]
2024/05/10 08:00:19 DEBUG : Creating backend with remote "/media/docker"
2024/05/10 08:00:19 DEBUG : Using config file from "/home/chris/.config/rclone/rclone.conf"
2024/05/10 08:00:19 DEBUG : Creating backend with remote "drive:Backup/Docker"
2024/05/10 08:00:19 DEBUG : Google drive root 'Backup/Docker': 'root_folder_id = 0AAxeANHaf86DUk9PVA' - save this in the config to speed up startup
2024/05/10 08:00:19 DEBUG : docker-compose.yml: Excluded (Path Filter)
2024/05/10 08:00:19 DEBUG : docker-compose.yml: Excluded
2024/05/10 08:00:19 DEBUG : appdata: Directory modification time the same (differ by -155.921µs, within tolerance 1ms)
2024/05/10 08:00:19 DEBUG : compose: Directory modification time the same (differ by -407.567µs, within tolerance 1ms)
2024/05/10 08:00:19 DEBUG : secrets: Directory modification time the same (differ by -93.745µs, within tolerance 1ms)
2024/05/10 08:00:19 DEBUG : rclone-filter.txt: Size and modification time the same (differ by -391.187µs, within tolerance 1ms)
2024/05/10 08:00:19 DEBUG : rclone-filter.txt: Unchanged skipping

rclone: make directory "appdata/audiobookshelf"?
y) Yes, this is OK (default)
n) No, skip this
s) Skip all make directory operations with no more questions
!) Do all make directory operations with no more questions
q) Exit rclone now.
y/n/s/!/q>

does it have something to do with the app directory existing on the remote already? the audiobookshelf directory isn't on the remote, however.

chelming · May 10, 2024, 12:22pm

$ cat slim.txt
+ ./rclone-filter.txt
- **

this seems to be working as I'd expect. but I'm still not sure why the other method didn't work. Does it create the directories regardless of whether it's syncing a file while it's on its way to search for any file named rclone-filter.txt? In my old example, shouldn't the appdata directory fail the first line, match on the second, and then not be created?

With my home assistant example, my filter looks like

- *.log
- *.log.*
- .git
- appdata/homeassistant/*
+ appdata/homeassistant/config/backups/**
- **

and my remote looks like

appdata/homeassistant/
├── config
│   ├── backups
│   │   │   ├── backup1.tar
│   │   │   ├── backup2.tar
│   ├── blueprints
│   │   ├── automation
│   │   │   ├── Blackshome
│   │   │   ├── EPMatt
│   │   │   ├── Mati24
│   │   │   ├── Raukze
│   │   │   ├── SgtBatten
│   │   │   ├── damru
│   │   │   ├── homeassistant
│   │   │   ├── hunterjm
│   │   │   ├── rohankapoorcom
│   │   │   └── sbyx
│   │   └── script
│   │       └── homeassistant
│   ├── custom_components
│   │   ├── __pycache__
│   │   ├── adaptive_lighting
│   │   │   ├── __pycache__
│   │   │   └── translations
│   │   ├── browser_mod
│   │   ├── envisalink_new
│   │   │   ├── __pycache__
│   │   │   ├── pyenvisalink
│   │   │   └── translations
│   │   ├── eventsensor
│   │   │   └── translations
│   │   ├── frigate
│   │   │   ├── __pycache__
│   │   │   └── translations

Is it correct that all of those sibling directories are getting created? and, honestly, looking at your example. I'm not even sure why that worked . Shouldn't it have stopped at the first home assistant line and never gotten to the second?

chelming · May 10, 2024, 1:52pm

ok maybe I'm an idiot and the correct config of the + before the - works

Animosity022 · May 10, 2024, 2:06pm

Filter process top down and exit on a match.

Generally, you want your most restrictive at the top and slowly work your way down.

So if rule 1 matches, it exits. If rule 1, 2, 3 don't match but 4 does, 4 matches and exits.

I usually trying to figure what I want to happen and explain that and rules can usually be crafted around that.

chelming · May 10, 2024, 3:30pm

here's the part I don't understand though, this doesn't work and copies all folders but not the files in the folders (I don't want the folders or the files). I was expecting that it would only create the directory tree where "rclone-filter.txt" exists

$ cat slim.txt
+ rclone-filter.txt
+ appdata/homeassistant/config/backups/**
- appdata/homeassistant/*
- **

Snippet from the sync run:

$ rclone sync --update --verbose --transfers 30 --checkers 8 --contimeout 60s --timeout 300s --retries 3 --low-level-retries 10 --filter-from /media/docker/slim.txt /media/docker "drive:Backup/Docker" --dump filters
2024/05/10 11:28:25 NOTICE: Automatically setting -vv as --dump is enabled
--- start filters ---
--- File filter rules ---
+ (^|/)rclone-filter\.txt$
+ (^|/)appdata/homeassistant/config/backups/.*$
- (^|/)appdata/homeassistant/[^/]*$
- (^|/).*$
--- Directory filter rules ---
+ ^.*$
+ (^|/)appdata/homeassistant/config/backups/.*/$
+ (^|/)appdata/homeassistant/config/backups/$
+ (^|/)appdata/homeassistant/config/$
+ (^|/)appdata/homeassistant/$
+ (^|/)appdata/$
+ (^|/)appdata/homeassistant/config/backups/.*$
- (^|/).*$
--- end filters ---
2024/05/10 11:28:25 DEBUG : rclone: Version "v1.66.0" starting with parameters ["rclone" "sync" "--update" "--verbose" "--transfers" "30" "--checkers" "8" "--contimeout" "60s" "--timeout" "300s" "--retries" "3" "--low-level-retries" "10" "--filter-from" "/media/docker/slim.txt" "/media/docker" "drive:Backup/Docker" "--dump" "filters"]
2024/05/10 11:28:25 DEBUG : Creating backend with remote "/media/docker"
2024/05/10 11:28:25 DEBUG : Using config file from "/home/chris/.config/rclone/rclone.conf"
2024/05/10 11:28:25 DEBUG : Creating backend with remote "drive:Backup/Docker"
2024/05/10 11:28:26 DEBUG : Google drive root 'Backup/Docker': 'root_folder_id = 0AAxeANHaf86DUk9PVA' - save this in the config to speed up startup
2024/05/10 11:28:26 DEBUG : appdata: Directory modification time the same (differ by -155.921µs, within tolerance 1ms)
2024/05/10 11:28:26 DEBUG : compose: Directory modification time the same (differ by -407.567µs, within tolerance 1ms)
2024/05/10 11:28:26 DEBUG : secrets: Directory modification time the same (differ by -93.745µs, within tolerance 1ms)
2024/05/10 11:28:26 DEBUG : rclone-filter.txt: Sizes identical
2024/05/10 11:28:26 DEBUG : rclone-filter.txt: Destination mod time is within 1ms of source and files identical, skipping
2024/05/10 11:28:26 DEBUG : appdata/jackett: Making directory with metadata
2024/05/10 11:28:27 INFO  : appdata/jackett: Made directory with metadata (mtime=2024-05-07T21:52:01.438002383-04:00)
2024/05/10 11:28:27 DEBUG : appdata/jellyfin: Making directory with metadata
2024/05/10 11:28:28 INFO  : appdata/jellyfin: Made directory with metadata (mtime=2024-04-28T20:22:07.295661416-04:00)
2024/05/10 11:28:28 DEBUG : appdata/komga: Making directory with metadata
2024/05/10 11:28:29 INFO  : appdata/komga: Made directory with metadata (mtime=2024-05-10T10:41:55.75937599-04:00)
2024/05/10 11:28:29 DEBUG : appdata/mosquitto: Making directory with metadata

Why is ^.*$ getting added as a rule? to try and find rclone-filter.txt in any directory? If I change that to - rclone-filter.txt it doesn't create that rule.

--- start filters ---
--- File filter rules ---
- (^|/)rclone-filter\.txt$
+ (^|/)appdata/homeassistant/config/backups/.*$
- (^|/)appdata/homeassistant/[^/]*$
- (^|/).*$
--- Directory filter rules ---
+ (^|/)appdata/homeassistant/config/backups/.*/$
+ (^|/)appdata/homeassistant/config/backups/$
+ (^|/)appdata/homeassistant/config/$
+ (^|/)appdata/homeassistant/$
+ (^|/)appdata/$
+ (^|/)appdata/homeassistant/config/backups/.*$
- (^|/).*$
--- end filters ---

This, on the other hand, works as expected:

$ cat slim.txt
+ /rclone-filter.txt
+ appdata/homeassistant/config/backups/**
- appdata/homeassistant/*
- **

Snippet from the sync run:

$ rclone sync --update --verbose --transfers 30 --checkers 8 --contimeout 60s --timeout 300s --retries 3 --low-level-retries 10 --filter-from /media/docker/slim.txt /media/docker "drive:Backup/Docker" --dump filters
2024/05/10 11:30:36 NOTICE: Automatically setting -vv as --dump is enabled
--- start filters ---
--- File filter rules ---
+ (^|/)\./rclone-filter\.txt$
+ (^|/)appdata/homeassistant/config/backups/.*$
- (^|/)appdata/homeassistant/[^/]*$
- (^|/).*$
--- Directory filter rules ---
+ (^|/)\./$
+ (^|/)appdata/homeassistant/config/backups/.*/$
+ (^|/)appdata/homeassistant/config/backups/$
+ (^|/)appdata/homeassistant/config/$
+ (^|/)appdata/homeassistant/$
+ (^|/)appdata/$
+ (^|/)appdata/homeassistant/config/backups/.*$
- (^|/).*$
--- end filters ---
2024/05/10 11:30:36 DEBUG : rclone: Version "v1.66.0" starting with parameters ["rclone" "sync" "--update" "--verbose" "--transfers" "30" "--checkers" "8" "--contimeout" "60s" "--timeout" "300s" "--retries" "3" "--low-level-retries" "10" "--filter-from" "/media/docker/slim.txt" "/media/docker" "drive:Backup/Docker" "--dump" "filters"]
2024/05/10 11:30:36 DEBUG : Creating backend with remote "/media/docker"
2024/05/10 11:30:36 DEBUG : Using config file from "/home/chris/.config/rclone/rclone.conf"
2024/05/10 11:30:36 DEBUG : Creating backend with remote "drive:Backup/Docker"
2024/05/10 11:30:36 DEBUG : Google drive root 'Backup/Docker': 'root_folder_id = 0AAxeANHaf86DUk9PVA' - save this in the config to speed up startup
2024/05/10 11:30:37 DEBUG : compose: Excluded
2024/05/10 11:30:37 DEBUG : secrets: Excluded
2024/05/10 11:30:37 DEBUG : rclone-filter.txt: Excluded (Path Filter)
2024/05/10 11:30:37 DEBUG : rclone-filter.txt: Excluded
2024/05/10 11:30:37 DEBUG : compose: Excluded
2024/05/10 11:30:37 DEBUG : secrets: Excluded
2024/05/10 11:30:37 DEBUG : appdata: Directory modification time the same (differ by -155.921µs, within tolerance 1ms)
2024/05/10 11:30:37 DEBUG : appdata/bazarr: Excluded
2024/05/10 11:30:37 DEBUG : appdata/audiobookshelf: Excluded
2024/05/10 11:30:37 DEBUG : appdata/cups: Excluded
2024/05/10 11:30:37 DEBUG : appdata/esphome: Excluded
2024/05/10 11:30:37 DEBUG : appdata/projectsend: Excluded
2024/05/10 11:30:37 DEBUG : appdata/traefik: Excluded
2024/05/10 11:30:37 DEBUG : appdata/unifi: Excluded
2024/05/10 11:30:37 DEBUG : appdata/zigbee2mqtt: Excluded
2024/05/10 11:30:37 DEBUG : appdata/frigate: Excluded
2024/05/10 11:30:37 DEBUG : appdata/komga: Excluded
2024/05/10 11:30:37 DEBUG : appdata/jackett: Excluded
2024/05/10 11:30:37 DEBUG : appdata/jellyfin: Excluded
2024/05/10 11:30:37 DEBUG : appdata/plex: Excluded
2024/05/10 11:30:37 DEBUG : appdata/mosquitto: Excluded
2024/05/10 11:30:37 DEBUG : appdata/mylar: Excluded
2024/05/10 11:30:37 DEBUG : appdata/overseerr: Excluded
2024/05/10 11:30:37 DEBUG : appdata/tautulli: Excluded
2024/05/10 11:30:37 DEBUG : appdata/syncthing: Excluded
2024/05/10 11:30:37 DEBUG : appdata/radarr: Excluded
2024/05/10 11:30:37 DEBUG : appdata/sonarr: Excluded
2024/05/10 11:30:37 DEBUG : appdata/readarr: Excluded
2024/05/10 11:30:37 DEBUG : appdata/sabnzbd: Excluded
2024/05/10 11:30:37 DEBUG : appdata/transmission: Excluded
2024/05/10 11:30:37 DEBUG : appdata/vaultwarden: Excluded
2024/05/10 11:30:37 DEBUG : appdata/snippet-box: Excluded
2024/05/10 11:30:37 DEBUG : appdata/zwavejs2mqtt: Excluded
2024/05/10 11:30:37 DEBUG : appdata/homeassistant: Directory modification time the same (differ by -535.018µs, within tolerance 1ms)
2024/05/10 11:30:37 DEBUG : appdata/homeassistant/db: Excluded
2024/05/10 11:30:37 DEBUG : appdata/homeassistant/config: Directory modification time the same (differ by -31.856µs, within tolerance 1ms)
2024/05/10 11:30:37 DEBUG : appdata/homeassistant/config/.cloud: Excluded
2024/05/10 11:30:37 DEBUG : appdata/homeassistant/config/.storage: Excluded
2024/05/10 11:30:37 DEBUG : appdata/homeassistant/config/blueprints: Excluded
2024/05/10 11:30:37 DEBUG : appdata/homeassistant/config/custom_components: Excluded
2024/05/10 11:30:37 DEBUG : appdata/homeassistant/config/deps: Excluded
2024/05/10 11:30:37 DEBUG : appdata/homeassistant/config/pip-uninstall-d_ah98t9: Excluded
2024/05/10 11:30:37 DEBUG : appdata/homeassistant/config/pip-uninstall-iwzpzyll: Excluded
2024/05/10 11:30:37 DEBUG : appdata/homeassistant/config/pip-uninstall-lecdnc3f: Excluded
2024/05/10 11:30:37 DEBUG : appdata/homeassistant/config/settings: Excluded
2024/05/10 11:30:37 DEBUG : appdata/homeassistant/config/tensorflow: Excluded
2024/05/10 11:30:37 DEBUG : appdata/homeassistant/config/themes: Excluded
2024/05/10 11:30:37 DEBUG : appdata/homeassistant/config/tts: Excluded
2024/05/10 11:30:37 DEBUG : appdata/homeassistant/config/www: Excluded
2024/05/10 11:30:37 DEBUG : appdata/homeassistant/config/packages: Excluded
2024/05/10 11:30:37 DEBUG : appdata/homeassistant/config/nest: Excluded
2024/05/10 11:30:37 DEBUG : appdata/homeassistant/config/scripts: Excluded
2024/05/10 11:30:37 DEBUG : appdata/homeassistant/config/backups: Directory modification time the same (differ by -413.722µs, within tolerance 1ms)
2024/05/10 11:30:38 DEBUG : Google drive root 'Backup/Docker': Waiting for checks to finish
2024/05/10 11:30:38 DEBUG : appdata/homeassistant/config/backups/4c5effda.tar: Sizes identical
2024/05/10 11:30:38 DEBUG : appdata/homeassistant/config/backups/4c5effda.tar: Destination mod time is within 1ms of source and files identical, skipping
2024/05/10 11:30:38 DEBUG : appdata/homeassistant/config/backups/f9e42559.tar: Sizes identical
2024/05/10 11:30:38 DEBUG : appdata/homeassistant/config/backups/f9e42559.tar: Destination mod time is within 1ms of source and files identical, skipping
2024/05/10 11:30:38 DEBUG : Google drive root 'Backup/Docker': Waiting for transfers to finish
2024/05/10 11:30:38 DEBUG : Waiting for deletions to finish
2024/05/10 11:30:38 INFO  : There was nothing to transfer

Animosity022 · May 10, 2024, 3:50pm

Ignore the filter for a second. What do you want to happen? Can you just explain your use case?

rclone.filter.txt just filters for that file only and does nothing else.

chelming · May 10, 2024, 3:57pm

assuming this structure:

.
├── appdata
│   ├── audiobookshelf
│   ├── bazarr
│   │   └── rclone-filter.txt
│   ├── cups
│   ├── esphome
│   ├── frigate
│   ├── homeassistant
│   │   └── config
│   │       ├── automations.yaml
│   │       └── backups
│   │           ├── 4c5effda.tar
│   │           └── f9e42559.tar
│   ├── jackett
│   ├── jellyfin
│   ├── komga
│   └── zwavejs2mqtt
├── compose
├── secrets
├── docker-compose.yml
├── rclone-filter.txt
├── secrets
└── slim.txt

I want the two tar files in appdata/homeassistant/config/backups and any file named rclone-filter.txt, one is currently at the root of the directory I'm copying and one is in appdata/bazarr. More might be added in the future. My expectation is that my remote will contain: ./rclone-filter.txt, appdata/bazarr/rclone-filter.txt, and the appdata/homeassistant/config/backups/*, but not any of the other folders in appdata.

Shouldn't this work?

+ rclone-filter.txt
+ appdata/homeassistant/config/backups/**
- appdata/homeassistant/*
- **

asdffdsa · May 10, 2024, 4:31pm

why not use --include-from=include.txt

and the contents of include.txt would be something like

rclone-filter.txt
/appdata/homeassistant/config/backups/*.tar

chelming · May 10, 2024, 4:33pm

Because ultimately, I'm going to be adding a lot more

+ *.yml
+ *.zip
- appdata/esphome/.esphome/**
+ appdata/esphome/*.yml
- appdata/mylar/cache/**
- appdata/mylar/app/*
- appdata/mylar/logs/**
+ appdata/mylar/**

Those first two are adding a + ^.*$ directory filter rule that is grabbing every single directory regardless of whether the files I want exist in them.

kapitainsky · May 10, 2024, 4:43pm

Ok - I think you have stumbled upon one of the side effects introduced by v1.66 new feature - directories metadata preservation.

It has been slowly worked on to rectify. For filters everything works again as it should in the latest beta.

If you do not want to use beta then roll back to 1.65.2 temporarily. Not sure if these changes will be included in v1.66.1 or only in v1.67

$ tree .
.
├── appdata
│   ├── audiobookshelf
│   ├── bazarr
│   │   └── rclone-filter.txt
│   ├── cups
│   │   └── xxxx
│   ├── esphome
│   └── homeassistant
│       └── config
│           ├── automations.yaml
│           └── backups
│               ├── 4c5effda.tar
│               └── f9e42559.tar
├── compose
├── docker-compose.yml
├── rclone-filter.txt
├── secrets
└── slim.txt

# rclone v1.66
$ rclone sync . drive:test --filter "+ rclone-filter.txt" --filter "- *" --dry-run
2024/05/10 18:50:01 NOTICE: appdata: Skipped make directory as --dry-run is set
2024/05/10 18:50:01 NOTICE: compose: Skipped make directory as --dry-run is set
2024/05/10 18:50:01 NOTICE: secrets: Skipped make directory as --dry-run is set
2024/05/10 18:50:01 NOTICE: rclone-filter.txt: Skipped copy as --dry-run is set (size 0)
2024/05/10 18:50:01 NOTICE: appdata/audiobookshelf: Skipped make directory as --dry-run is set
2024/05/10 18:50:01 NOTICE: appdata/bazarr: Skipped make directory as --dry-run is set
2024/05/10 18:50:01 NOTICE: appdata/cups: Skipped make directory as --dry-run is set
2024/05/10 18:50:01 NOTICE: appdata/esphome: Skipped make directory as --dry-run is set
2024/05/10 18:50:01 NOTICE: appdata/homeassistant: Skipped make directory as --dry-run is set
2024/05/10 18:50:01 NOTICE: appdata/homeassistant/config: Skipped make directory as --dry-run is set
2024/05/10 18:50:01 NOTICE: appdata/bazarr/rclone-filter.txt: Skipped copy as --dry-run is set (size 0)
2024/05/10 18:50:01 NOTICE: appdata/homeassistant/config/backups: Skipped make directory as --dry-run is set
2024/05/10 18:50:01 NOTICE:
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Transferred:            2 / 2, 100%
Elapsed time:         1.3s


# rclone v1.67 beta
$ rclone-beta sync . drive:test --filter "+ rclone-filter.txt" --filter "- *" --dry-run
2024/05/10 18:50:33 NOTICE: rclone-filter.txt: Skipped copy as --dry-run is set (size 0)
2024/05/10 18:50:33 NOTICE: appdata/bazarr/rclone-filter.txt: Skipped copy as --dry-run is set (size 0)
2024/05/10 18:50:33 NOTICE: appdata/bazarr: Skipped set directory modification time as --dry-run is set
2024/05/10 18:50:33 NOTICE: appdata: Skipped set directory modification time as --dry-run is set
2024/05/10 18:50:33 NOTICE:
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Transferred:            2 / 2, 100%
Elapsed time:         1.2s

Animosity022 · May 10, 2024, 4:50pm

The 3rd line isn't relevant as you already exclude with the 4th line.

Instead of the copy, just test with rclone ls.

chelming · May 10, 2024, 5:10pm

Thank you so much! The 1.67 beta does indeed fix everything for me. I've been banging my head against the wall for the past two days!

system · May 13, 2024, 5:11pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.