Why does rclone parse/list content of directories that are excluded (filter) on a sync command? In my mind it should just ignore the directory, both on the source and destination.
I.e I have a --filter-from that includes - /some/dir/path/something.sparsebundle/**
Why does rclone use resources/time to list what's inside the excluded directory, and then match it?
and move on to the next (instead of listing/comparing the thousands of files that are there)?
My use case is a large dataset with many files of different sizes, but with a more or less fixed folder structure. It's all ready synced 1000 of times, but has now become big enough that I need to take measures to cut down the time (and some dropbox errors error reading source directory: <!DOCTYPE html>).
One way (or at least I thought) would be to just use regex to ignore a "path type" to limit the amount of files rclone chews through.
Did not really think they are relevant to this question. Not even the version (with in reason) should be important here – this is more of a "design" / feature question than anything.
If you think any specific information will help to expand on my problem/question, I'm happy to contribute.
Rclone should be doing this for the local backend at least.
If you provide more info such as full command line, redacted config, rclone version , contents of filter from file, we can have a go at working out why it isn't.
Fast list uses a different API which recursively lists everything under a directory. There is no option to that API to tell it to ignore directories.
This section in the docs should probably note that if you are using filters then --fast-list will recurse all directories and filter afterwards which may be less efficient.
Yes, that I get, but shouldn't the filters be respected when rclone is parsing the list from the --fast-list API call? i.e ignoring all files/dirs inside an ignored directory?
I think this is working as designed. The files were excluded like the filter says.
- ^some/dir/path/something.sparsebundle/.*$
In rclone speak that filter says exclude any files which start with some/dir/path/something.sparsebundle/. Rclone filters only refer to files, not directories - the fact that they cause directories to be skipped when not using --fast-list is an optimization.
Using --fast-list causes the skipping directory optimization not to be available hence the listing of each file as Excluded.