Filter a folder and all sub folders/files not working

notDavid · June 25, 2019, 10:28am

Hi there,

i'm trying to exclude the .AppleDouble folder and all sub-folders and files inside it. This .AppleDouble folder can exist multiple times in all subfolders in the tree.

This is the filter i tried (in a file):

- .AppleDouble/**
- **/.AppleDouble/**

But rclone --filter-from ~/.rclone.filter copy /backups/ remote: still copies

2019-06-25 12:12:09 INFO : data/.AppleDouble/.Parent: Copied (new)

so the filter does not work.

Please advise
Thank you!

ncw · June 25, 2019, 2:38pm

This one is correct - you don't need the other one.

I tried your filter exactly as-is and it worked fine for me

$ rclone ls .
        0 a
        0 .AppleDouble/file.txt
$ rclone ls . --filter-from ../filter 
        0 a
$ cat ../filter 
- .AppleDouble/**
- **/.AppleDouble/**

Which version of rclone are you using (rclone version).

Can you also add --dump filters -vv to the command line and report back what the filters dump as?

notDavid · June 25, 2019, 4:11pm

Thanks for your help, here's the log:


pi@pi2:~/tmp $ rclone version
rclone v1.48.0
- os/arch: linux/arm
- go version: go1.12.6
pi@pi2:~/tmp $ rclone -vPL --dump filters -vv --stats-file-name-length 0 --filter-from ~/.rclone.filter --bwlimit=1500 --transfers 1 --includ
e "/data/**" copy . downloads:test/
--- start filters ---
--- File filter rules ---
+ ^data/.*$
- (^|/)\.DS_Store$
- (^|/)\.AppleDouble/.*$
- (^|/).*/\.AppleDouble/.*$
- ^.*$
--- Directory filter rules ---
+ ^data/.*/$
+ ^data/$
+ ^data/.*$
- (^|/)\.AppleDouble/.*$
- (^|/).*/\.AppleDouble/.*$
- ^.*$
--- end filters ---
2019/06/25 17:01:53 DEBUG : rclone: Version "v1.48.0" starting with parameters ["rclone" "-vPL" "--dump" "filters" "-vv" "--stats-file-name-l
ength" "0" "--filter-from" "/home/pi/.rclone.filter" "--bwlimit=1500" "--transfers" "1" "--include" "/data/**" "copy" "." "downloads:test/"]
2019/06/25 17:01:53 DEBUG : Using config file from "/home/pi/.rclone.conf"
2019/06/25 17:01:53 INFO  : Starting bandwidth limiter at 1.465MBytes/s
2019/06/25 17:01:54 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded)
2019/06/25 17:01:54 DEBUG : pacer: Rate limited, increasing sleep to 1.958669983s
2019/06/25 17:01:54 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded)
2019/06/25 17:01:54 DEBUG : pacer: Rate limited, increasing sleep to 2.590264334s
2019/06/25 17:01:56 DEBUG : pacer: Reducing sleep to 0s
2019-06-25 17:02:00 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded)
2019-06-25 17:02:00 DEBUG : pacer: Rate limited, increasing sleep to 1.315141311s
2019-06-25 17:02:00 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded)
2019-06-25 17:02:00 DEBUG : pacer: Rate limited, increasing sleep to 2.112119057s
2019-06-25 17:02:02 DEBUG : pacer: low level retry 3/10 (error googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded)
2019-06-25 17:02:02 DEBUG : pacer: Rate limited, increasing sleep to 4.689154913s
2019-06-25 17:02:04 DEBUG : pacer: low level retry 4/10 (error googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded)
2019-06-25 17:02:04 DEBUG : pacer: Rate limited, increasing sleep to 8.059753494s
2019-06-25 17:02:09 DEBUG : pacer: Reducing sleep to 0s
2019-06-25 17:02:09 INFO  : Encrypted drive 'downloads:test/': Waiting for checks to finish
2019-06-25 17:02:09 INFO  : Encrypted drive 'downloads:test/': Waiting for transfers to finish
2019-06-25 17:02:20 DEBUG : 37mo1mnmjoi9g63tt49j9re2vo/411al0c24pa0rqe5m5cbk8s79g: Sending chunk 0 length 8388608
2019-06-25 17:02:23 DEBUG : 37mo1mnmjoi9g63tt49j9re2vo/411al0c24pa0rqe5m5cbk8s79g: Sending chunk 8388608 length 6123646
2019-06-25 17:02:28 INFO  : data/backup.zip: Copied (new)
2019-06-25 17:02:32 INFO  : data/.AppleDouble/.Parent: Copied (new)
2019-06-25 17:02:32 DEBUG : pacer: low level retry 1/1 (error googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded)
2019-06-25 17:02:32 DEBUG : pacer: Rate limited, increasing sleep to 1.269745564s
2019-06-25 17:02:32 DEBUG : data/.AppleDouble/backup.zip: Received error: googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded - low
level retry 1/10
2019-06-25 17:02:32 DEBUG : pacer: Reducing sleep to 0s
2019-06-25 17:02:33 DEBUG : pacer: low level retry 1/1 (error googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded)
2019-06-25 17:02:33 DEBUG : pacer: Rate limited, increasing sleep to 1.443699482s
2019-06-25 17:02:33 DEBUG : data/.AppleDouble/backup.zip: Received error: googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded - low
level retry 2/10
2019-06-25 17:02:33 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded)
2019-06-25 17:02:33 DEBUG : pacer: Rate limited, increasing sleep to 2.637273491s
2019-06-25 17:02:35 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded)
2019-06-25 17:02:35 DEBUG : pacer: Rate limited, increasing sleep to 4.12625937s
2019-06-25 17:02:38 DEBUG : pacer: Reducing sleep to 0s
2019-06-25 17:02:42 DEBUG : pacer: low level retry 1/1 (error googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded)
2019-06-25 17:02:42 DEBUG : pacer: Rate limited, increasing sleep to 1.760807413s
2019-06-25 17:02:42 DEBUG : data/.AppleDouble/backup.zip: Received error: googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded - low
level retry 3/10
2019-06-25 17:02:42 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded)
2019-06-25 17:02:42 DEBUG : pacer: Rate limited, increasing sleep to 2.797446636s
2019-06-25 17:02:43 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded)
2019-06-25 17:02:43 DEBUG : pacer: Rate limited, increasing sleep to 4.372370698s
2019-06-25 17:02:46 DEBUG : pacer: Reducing sleep to 0s
2019-06-25 17:02:51 DEBUG : pacer: low level retry 1/1 (error googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded)
2019-06-25 17:02:51 DEBUG : pacer: Rate limited, increasing sleep to 1.383873811s
2019-06-25 17:02:51 DEBUG : data/.AppleDouble/backup.zip: Received error: googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded - low
level retry 4/10
2019-06-25 17:02:51 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded)
2019-06-25 17:02:51 DEBUG : pacer: Rate limited, increasing sleep to 2.322750439s
2019-06-25 17:02:52 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded)
2019-06-25 17:02:52 DEBUG : pacer: Rate limited, increasing sleep to 4.803954771s
2019-06-25 17:02:54 DEBUG : pacer: low level retry 3/10 (error googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded)
2019-06-25 17:02:54 DEBUG : pacer: Rate limited, increasing sleep to 8.410518491s
2019-06-25 17:03:00 DEBUG : pacer: Reducing sleep to 0s
2019-06-25 17:03:09 INFO  : data/.AppleDouble/backup.zip: Copied (new)
Transferred:       13.844M / 13.844 MBytes, 100%, 184.849 kBytes/s, ETA 0s
Errors:                 0
Checks:                 0 / 0, -
Transferred:            3 / 3, 100%
Elapsed time:     1m16.6s
2019/06/25 17:03:09 INFO  :
Transferred:       13.844M / 13.844 MBytes, 100%, 184.848 kBytes/s, ETA 0s
Errors:                 0
Checks:                 0 / 0, -
Transferred:            3 / 3, 100%
Elapsed time:     1m16.6s

2019/06/25 17:03:09 DEBUG : 8 go routines active
2019/06/25 17:03:09 DEBUG : rclone: Version "v1.48.0" finishing with parameters ["rclone" "-vPL" "--dump" "filters" "-vv" "--stats-file-name-
length" "0" "--filter-from" "/home/pi/.rclone.filter" "--bwlimit=1500" "--transfers" "1" "--include" "/data/**" "copy" "." "downloads:test/"]
pi@pi2:~/tmp $ cat /home/pi/.rclone.filter
# https://rclone.org/filtering/#filter-from-read-filtering-patterns-from-a-file
- .DS_Store
- .AppleDouble/**
- **/.AppleDouble/**
pi@pi2:~/tmp $

I hope i'm not making some really obvious mistake

Animosity022 · June 25, 2019, 4:20pm

The order of the filters matter as you are including everything before excluding it.

Can you share your filter file?

notDavid · June 25, 2019, 4:55pm

Thanks for your reply, the filter file is also in the code i pasted, all the way at the bottom; It's this:

pi@pi2:~/tmp $ cat /home/pi/.rclone.filter
# https://rclone.org/filtering/#filter-from-read-filtering-patterns-from-a-file
- .DS_Store
- .AppleDouble/**
- **/.AppleDouble/**

Animosity022 · June 25, 2019, 5:09pm

You need to use the filter or the include and not both.

The include data is taking precedence as you are including that.

Adjust the filter to be:

# https://rclone.org/filtering/#filter-from-read-filtering-patterns-from-a-file
- .DS_Store
- .AppleDouble/**
- **/.AppleDouble/**
+ /data/**

I believe that is what you are trying to accomplish.

notDavid · June 25, 2019, 5:20pm

Ahhh okay i understand.

I put the --filter-from ~/.rclone.filter in a default rclone "fish abbreviation" (something like a Bash alias) because i always want to exclude those, regardless of what i copy.

However that won't work in combination with --include...

Thank you!

Animosity022 · June 25, 2019, 5:23pm

Hmm, I wonder if a flag or if it should error out if you try that combination together.

It just creates some funky results if you try to mix both as you can see from the filter dump, the include jumped to the top of the filter list.

notDavid · June 25, 2019, 5:55pm

I think it works as documented here

The only "issue" is, if you want to permanently exclude files like these, you're out of luck, because the --filter-from always has the lowest priority

(If i understand correctly.)

Animosity022 · June 25, 2019, 5:57pm

I'm not sure what the issue is you are describing. You should be able to logically put the filter in order to include / exclude what you need. Is there another case you are trying to solve?

notDavid · June 25, 2019, 6:05pm

What i'd want is to make the --filter-from ~/.rclone.filter "permanent", every time i run an rclone command, and give it the higher priority over any other include or exclude.

So i simply created something like an alias which says rclone = rclone --filter-from ~/.rclone.filter
So i don't have to type it all the time... but it will never work, because it will always have the lowest priority.

Animosity022 · June 25, 2019, 6:12pm

You can use environment variables in rclone to set things that stick.

filter from would be:

export RCLONE_FILTER_FROM=/home/felix/filter.txt

I think you are walking down a bit of a logic issues if you want to combine includes and excludes. If your goal is to exclude things, you can use the excludes to get rid of the apple doubles and whatnot.

notDavid · June 25, 2019, 9:03pm

I'm not sure what "RCLONE_FILTER_FROM" does exactly as i can't find anything in the docs. But i assume this doesn't change anything regarding the priority of --include / --exclude / --filter. Which means RCLONE_FILTER_FROM won't work when also using --include.

I don't see why there would be any logic issues... it's just a matter of "include/exclude/filters" and "priority" of each rule.
But the way rclone works right now, it's simply not possible to "pin certain exclude rules as highest priority".

Anyways... i understand how it works now, thanks for all the info!

system · September 23, 2019, 9:03pm

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.