Filters with inclusions and exclusions and permission errors

[jwink3101 (Justin Winokur)]

What is the problem you are having with rclone?

It does not seem to be respecting my filters as I would have expected. There are hidden files I want to keep and also hidden files and directories I do not care about. Some of them are not owned by me (system stuff like .Trashes) so I am trying to exclude them.

When I just exclude all hidden items, it is all fine. But when I try to include my desired hidden items (ending in .notes.yaml) It does include them but also throws IO errors from the permissions.

It shouldn't be trying to read those directories, right?

I also tried putting my include filter at the end but that, as I would expect, kills my other filter

My guess as to what is happening is that when there is the include filter, it is trying to list the hidden directories to match against but when that isn't there, it doesn't try to list them. Is that correct? That seems like incorrect behavior as it is not a bucket remote (it's local file system) and it can match the directory name

What is your rclone version (output from rclone version)

Which OS you are using and how many bits (eg Windows 7, 64 bit)

macOS

Which cloud storage system are you using? (eg Google Drive)

Local to crypt

The command you were trying to run (eg rclone copy /tmp remote:tmp)

This command correctly includes the files I want but does not exclude hidden directories

rclone --dry-run\
    --config backups/backup_sync.rclone.cfg \
    -v --progress \
    --filter "+ .*.notes.yaml" \
    --filter "- .*" \
    --filter "- .*/**" \
    sync . backup:

I would assume it is something to do with my excludes but when I just run

rclone --dry-run\
    --config backups/backup_sync.rclone.cfg \
    -v --progress \
    --filter "- .*" \
    --filter "- .*/**" \
    sync . backup:

it correctly excludes the other hidden directories

A log from the command with the -vv flag

When the first command is run:

(I notice certain files are excluded (e.g. .DS_Store) but the permission denied shouldn't have even happened given the filter)

...
2020/05/16 17:54:07 DEBUG : .DS_Store: Excluded
...
2020/05/16 17:54:41 ERROR : .DocumentRevisions-V100: failed to open directory ".DocumentRevisions-V100": open ****.DocumentRevisions-V100: permission denied
...

2020-05-16 17:54:55 ERROR : Attempt 3/3 failed with 4 errors and: not deleting files as there were IO errors

With the second command

...
2020/05/16 17:58:31 DEBUG : .test.notes.yaml: Excluded
2020/05/16 17:58:31 DEBUG : .DS_Store: Excluded
...

NO permission errors

---

I am not sure I understand why it is trying access excluded directories when there is the include (that doesn't match!) but not otherwise. Is that expected?

[calisro (Rob)]

I believe rclone still needs to match the files. So if you don't have access to those directories to match the files I don't think it'll exclude them. You can test this by creating a similar structure elsewhere with files owned by you and test it.

Directory matches are only used to optimise directory access patterns - you must still match the files that you want to match.

Also since this isn't anchored to the root it's possible that this matches something inside a directory you've excluded so rclone wants to look inside it.

.*.notes.yaml

Order matters.

If the files are in the root would this work for you?

--filter "+ /.*.notes.yaml" \
--filter "- .*" \
--filter "- .*/**" \

[jwink3101 (Justin Winokur)]

Ok. I see it now. When I just had the hidden files, it could optimize itself because there was nothing inside that could possibly match! But that's not the case with the other include. It's unfortunate since it throws all of those (non-fatal) errors and is less optimized. But it also 100% makes sense!

Thanks for the clarification. And the files aren't always in the root so that wouldn't work.

But I'll either live with the error or explicitly exclude the permission ones (I was trying to avoid that but oh well).

Thanks for the clarificaton.