Problem filtering filenames with filter-from

[Kyosfonica]

What is the problem you are having with rclone?

I am creating a script to copy files from one remote folder in my network to a folder in my computer then merge all those files into a single flat folder and then I process them with another tool.

(samba) Network-folder -> (rclone copy) temp-folder -> flat-folder
The issue is within the rclone copy step.

The issue I am having is that I am listing the filenames of the files that are already in the flat folder to prevent rclone from copy them again as these files are 100% the same in the network folder and I want to prevent to copy them again because those are very big files and I want to save this time.

As I undersand the issue, it seems that the characters in the filename makes the rclone filters to go crazy and cannot process them correctly. I looked for days on google and in this forum but I haven't found the issue. I am far from being an expert in regexp as this is handled by rclone but the name of the file is causing the rule to explode as I can see using https://regex101.com/ and checking the error there.

Run the command 'rclone version' and share the full output of the command.

E:\HD_convert>rclone version
rclone v1.60.1
- os/version: Microsoft Windows 11 Pro 22H2 (64 bit)
- os/kernel: 10.0.22621.963 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.19.3
- go/linking: static
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

All is local filesystem the network folder is a samba share

The command you were trying to run (eg rclone copy /tmp remote:tmp)

I am testing the filter first as recommendation from another user in the forum on another filter issue.

rclone ls --filter-from filter.txt videos

The rclone config contents with secrets removed.

Default config used, no remotes.

2022/12/16 08:15:37 NOTICE: Config file "C:\\Users\\kyos\\AppData\\Roaming\\rclone\\rclone.conf" not found - using defaults

A log from the command with the -vv flag

E:\HD_convert>rclone ls --filter-from filter.txt videos -vv
2022/12/16 08:48:31 Failed to load filters: bad glob pattern "Battlestar Galactica - El plan (2009) [BDRemux 1080p VC-1 ES DD 5.1 - EN DTS-HD MA 5.1 Subs][HDO].mkv" (regexp "(^|/)Battlestar Galactica - El plan \\(2009\\) [BDRemux 1080p VC-1 ES DD 5.1 - EN DTS-HD MA 5.1 Subs][HDO]\\.mkv$"): error parsing regexp: invalid character class range: `C-1`

The content of the filter right now is:

+ *.mkv
- Battlestar Galactica - El plan (2009) [BDRemux 1080p VC-1 ES DD 5.1 - EN DTS-HD MA 5.1 Subs][HDO].mkv
- **

I suspect the issue is with the VC-1 and the DTS-HD portions of the name but this is the filename. How can I escape those to still make the filename to match?
I tried with \ and the filter parses the file but then it doesn't match and the file gets listed

Help and feedback are more than appreciated

Cheers

[Animosity022]

The first bit would to fix the ordering as you want the - before the + as the filtering is down by top down matching.

So if you flag that first match, it never hits the second anyway.

Next, you are correct as you have filtering special characters in there.

Rclone Filtering

Guessing that is Windows as you've got a snippet of the rclone version but looks like a Windows prompt there.

For any special characters that you specifically want to match, you'd want to escape them with a backslash \ before the special character.

So a [ would look like [

Here is how it would look:

felix@gemini:~$ rclone ls GD: --filter-from filter
2022/12/16 07:44:25 Failed to load filters: bad glob pattern "Battlestar Galactica - El plan (2009) [BDRemux 1080p VC-1 ES DD 5.1 - EN DTS-HD MA 5.1 Subs][HDO].mkv" (regexp "(^|/)Battlestar Galactica - El plan \\(2009\\) [BDRemux 1080p VC-1 ES DD 5.1 - EN DTS-HD MA 5.1 Subs][HDO]\\.mkv$"): error parsing regexp: invalid character class range: `C-1`
felix@gemini:~$ cat filter
- Battlestar Galactica - El plan (2009) [BDRemux 1080p VC-1 ES DD 5.1 - EN DTS-HD MA 5.1 Subs][HDO].mkv
+ *.mkv
- **
felix@gemini:~$ vi filter
felix@gemini:~$
felix@gemini:~$ rclone ls GD: --filter-from filter
felix@gemini:~$ cat filter
- Battlestar Galactica - El plan \(2009\) \[BDRemux 1080p VC-1 ES DD 5.1 - EN DTS-HD MA 5.1 Subs\]\[HDO\].mkv
+ *.mkv
- **

[Kyosfonica]

First of all, thanks for the response and the time you have dedicated to me. I will always be grateful for people that help others.

I modified the issue to show more information about what you correctly supposed.

I understood the documentation wrong regarding the order of the filtering. I fixed that as well.

I testesd your solution and now it works

This is how I was escaping before. It was my fault not to put that in the issue as well.

- Battlestar Galactica \- El plan \(2009\) \[BDRemux 1080p VC\-1 ES DD 5.1 \- EN DTS\-HD MA 5.1 Subs\]\[HDO\].mkv

So, if I understand correctly what I have to escape are (, ), [, ] but not the - nor the dots.

Is that correct?

[Animosity022]

That seems right to me.

Rclone regex's are a little different as it's a dot normally matches one character, but I don't think so here.