Problem with --exclude-from flag

fidodone · October 20, 2021, 3:46am

What is the problem you are having with rclone?

I'm having trouble with some copies in rclone
When I copy using the flag --exclude-from and the file has some line with special characters type '[or]' the rclone returns the following error:
Failed to load filters: bad glob pattern "file (with special characters)" (regexp "(^|/)file (with special characters)"): error parsing regexp: invalid character class range: L-1

What is your rclone version (output from `rclone version`)

rclone v1.56.2

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone copy tdrive$TDN:folder_a tdrive$TDN:folder_b -P -v --exclude-from exclude.txt --max-transfer 700G --drive-stop-on-upload-limit

The rclone config contents with secrets removed.

[tdrive1]
type = drive
client_id = id
client_secret = secret
scope = drive
service_account_file = json.json
team_drive = team_drive_id
root_folder_id =

A log from the command with the `-vv` flag

2021/10/20 00:45:14 Failed to load filters: bad glob pattern "The.OA.S01.1080p.NF.WEB-DL.DDP5.1.H.264.DUAL-alfaHD/The OA - S01E01 - Chapter 1- Homecoming [WEBDL-1080p EAC3 x264-alfaHD].mkv" (regexp "(^|/)The\\.OA\\.S01\\.1080p\\.NF\\.WEB-DL\\.DDP5\\.1\\.H\\.264\\.DUAL-alfaHD/The OA - S01E01 - Chapter 1- Homecoming [WEBDL-1080p EAC3 x264-alfaHD]\\.mkv$"): error parsing regexp: invalid character class range: `L-1`

Animosity022 · October 20, 2021, 10:24am

Seems like you have an error in your exclude file but you didn't share the exclude file so not sure how we can help.

ivandeex · October 20, 2021, 11:29am

Actually it's a bug in rclone, discovered just recently - the filtering rules should be glob-based but leaked some regexp-patterns into pattern matcher.
See:

By 1.58 we hope to settle the syntax by explicitly isolating glob and regexp parts in filter rules, transforming the bug into documented feature.

For now you might mitigate it by \-escaping. You are welcome to report your findings in the ongoing task:

github.com/rclone/rclone

filter: add `{{ regexp }}` syntax to pattern matches - fixes #4074

rclone:master ← rclone:fix-4074-filter-regexp

opened 12:03PM - 09 Oct 21 UTC

ncw

+166 -13

There has been a desire from more advanced rclone users to have regexp filtering… as well as the glob filtering. This patch adds regexp filtering using this syntax `{{ regexp }}` which is currently a syntax error, so is backwards compatibile. This means regexps can be used everywhere globs can be used, and that they also can be mixed with globs in the same pattern, eg `*.{{jpe?g}}` See: #4074 for background Fixes #4074 @ivandeex and @albertony would appreciate your feedback on this - is this a good idea or not?

fidodone · October 20, 2021, 1:19pm

the exclusion list is generated from another rclone command

rclone lsf -R --files-only tdrive1:folder_b >> /home/luiz/transfered.txt
example of the contents of the list:

The.OA.S01.1080p.NF.WEB-DL.DDP5.1.H.264.DUAL-alfaHD/The OA - S01E01 - Chapter 1- Homecoming [WEBDL-1080p EAC3 x264-alfaHD].mkv
The.OA.S01.1080p.NF.WEB-DL.DDP5.1.H.264.DUAL-alfaHD/The OA - S01E02 - Chapter 2- New Colossus [WEBDL-1080p EAC3 x264-alfaHD].mkv
The.OA.S01.1080p.NF.WEB-DL.DDP5.1.H.264.DUAL-alfaHD/The OA - S01E03 - Chapter 3- Champion [WEBDL-1080p EAC3 x264-alfaHD].mkv
The.OA.S01.1080p.NF.WEB-DL.DDP5.1.H.264.DUAL-alfaHD/The OA - S01E04 - Chapter 4- Away [WEBDL-1080p EAC3 x264-alfaHD].mkv
The.OA.S01.1080p.NF.WEB-DL.DDP5.1.H.264.DUAL-alfaHD/The OA - S01E05 - Chapter 5- Paradise [WEBDL-1080p EAC3 x264-alfaHD].mkv
The.OA.S01.1080p.NF.WEB-DL.DDP5.1.H.264.DUAL-alfaHD/The OA - S01E06 - Chapter 6- Forking Paths [WEBDL-1080p EAC3 x264-alfaHD].mkv
The.OA.S01.1080p.NF.WEB-DL.DDP5.1.H.264.DUAL-alfaHD/The OA - S01E07 - Chapter 7- Empire of Light [WEBDL-1080p EAC3 x264-alfaHD].mkv
The.OA.S01.1080p.NF.WEB-DL.DDP5.1.H.264.DUAL-alfaHD/The OA - S01E08 - Chapter 8- Invisible Self [WEBDL-1080p EAC3 x264-alfaHD].mkv
The.OA.S02.1080p.NF.WEB-DL.DDP5.1.H.264.DUAL-alfaHD/The OA - S02E01 - Chapter 1- Angel of Death [WEBDL-1080p EAC3 x264-alfaHD].mkv
The.OA.S02.1080p.NF.WEB-DL.DDP5.1.H.264.DUAL-alfaHD/The OA - S02E01 - Chapter 1- Angel of Death WEBDL-1080p EAC3 x264-alfaHD.mkv
The.OA.S02.1080p.NF.WEB-DL.DDP5.1.H.264.DUAL-alfaHD/The OA - S02E02 - Chapter 2- Treasure Island [WEBDL-1080p EAC3 x264-alfaHD].mkv
The.OA.S02.1080p.NF.WEB-DL.DDP5.1.H.264.DUAL-alfaHD/The OA - S02E02 - Chapter 2- Treasure Island WEBDL-1080p EAC3 x264-alfaHD.mkv
The.OA.S02.1080p.NF.WEB-DL.DDP5.1.H.264.DUAL-alfaHD/The OA - S02E03 - Chapter 3- Magic Mirror [WEBDL-1080p EAC3 x264-alfaHD].mkv
The.OA.S02.1080p.NF.WEB-DL.DDP5.1.H.264.DUAL-alfaHD/The OA - S02E03 - Chapter 3- Magic Mirror WEBDL-1080p EAC3 x264-alfaHD.mkv
The.OA.S02.1080p.NF.WEB-DL.DDP5.1.H.264.DUAL-alfaHD/The OA - S02E04 - Chapter 4- SYZYGY [WEBDL-1080p EAC3 x264-alfaHD].mkv
The.OA.S02.1080p.NF.WEB-DL.DDP5.1.H.264.DUAL-alfaHD/The OA - S02E04 - Chapter 4- SYZYGY WEBDL-1080p EAC3 x264-alfaHD.mkv
The.OA.S02.1080p.NF.WEB-DL.DDP5.1.H.264.DUAL-alfaHD/The OA - S02E05 - Chapter 5- The Medium & The Engineer [WEBDL-1080p EAC3 x264-alfaHD].mkv
The.OA.S02.1080p.NF.WEB-DL.DDP5.1.H.264.DUAL-alfaHD/The OA - S02E05 - Chapter 5- The Medium & The Engineer WEBDL-1080p EAC3 x264-alfaHD.mkv
The.OA.S02.1080p.NF.WEB-DL.DDP5.1.H.264.DUAL-alfaHD/The OA - S02E06 - Chapter 6- Mirror Mirror [WEBDL-1080p EAC3 x264-alfaHD].mkv
The.OA.S02.1080p.NF.WEB-DL.DDP5.1.H.264.DUAL-alfaHD/The OA - S02E06 - Chapter 6- Mirror Mirror WEBDL-1080p EAC3 x264-alfaHD.mkv
The.OA.S02.1080p.NF.WEB-DL.DDP5.1.H.264.DUAL-alfaHD/The OA - S02E07 - Chapter 7- Nina Azarova [WEBDL-1080p EAC3 x264-alfaHD].mkv
The.OA.S02.1080p.NF.WEB-DL.DDP5.1.H.264.DUAL-alfaHD/The OA - S02E07 - Chapter 7- Nina Azarova WEBDL-1080p EAC3 x264-alfaHD.mkv
The.OA.S02.1080p.NF.WEB-DL.DDP5.1.H.264.DUAL-alfaHD/The OA - S02E08 - Chapter 8- Overview [WEBDL-1080p EAC3 x264-alfaHD].mkv
The.OA.S02.1080p.NF.WEB-DL.DDP5.1.H.264.DUAL-alfaHD/The OA - S02E08 - Chapter 8- Overview WEBDL-1080p EAC3 x264-alfaHD.mkv
The.Orville.S01.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD/The.Orville.S01E01.Old.Wounds.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD.mkv
The.Orville.S01.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD/The.Orville.S01E02.Command.Performance.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD.mkv
The.Orville.S01.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD/The.Orville.S01E03.About.A.Girl.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD.mkv
The.Orville.S01.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD/The.Orville.S01E04.If.The.Stars.Should.Appear.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD.mkv
The.Orville.S01.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD/The.Orville.S01E05.Pria.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD.mkv
The.Orville.S01.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD/The.Orville.S01E06.Krill.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD.mkv
The.Orville.S01.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD/The.Orville.S01E07.Majority.Rule.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD.mkv
The.Orville.S01.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD/The.Orville.S01E08.Into.The.Fold.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD.mkv
The.Orville.S01.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD/The.Orville.S01E09.Cupids.Dagger.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD.mkv
The.Orville.S01.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD/The.Orville.S01E10.Firestorm.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD.mkv
The.Orville.S01.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD/The.Orville.S01E11.New.Dimensions.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD.mkv
The.Orville.S01.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD/The.Orville.S01E12.Mad.Idolatry.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD.mkv
The.Orville.S02.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD/The.Orville.S02E01.Jaloja.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD.mkv
The.Orville.S02.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD/The.Orville.S02E02.Primal.Urges.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD.mkv
The.Orville.S02.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD/The.Orville.S02E03.Home.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD.mkv
The.Orville.S02.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD/The.Orville.S02E04.Nothing.Left.On.Earth.Excepting.Fishes.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD.mkv
The.Orville.S02.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD/The.Orville.S02E05.All.the.World.Is.Birthday.Cake.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD.mkv
The.Orville.S02.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD/The.Orville.S02E06.A.Happy.Refrain.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD.mkv
The.Orville.S02.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD/The.Orville.S02E07.Deflectors.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD.mkv
The.Orville.S02.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD/The.Orville.S02E08.Identity.Part.I.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD.mkv
The.Orville.S02.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD/The.Orville.S02E09.Identity.Part.II.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD.mkv
The.Orville.S02.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD/The.Orville.S02E10.Blood.of.Patriots.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD.mkv
The.Orville.S02.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD/The.Orville.S02E11.Lasting.Impressions.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD.mkv
The.Orville.S02.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD/The.Orville.S02E12.Sanctuary.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD.mkv
The.Orville.S02.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD/The.Orville.S02E13.Tomorrow.and.Tomorrow.and.Tomorrow.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD.mkv
The.Orville.S02.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD/The.Orville.S02E14.The.Road.Not.Taken.1080p.DSNP.WEB-DL.DDP5.1.H.264.DUAL-alfaHD.mkv

fidodone · October 20, 2021, 1:22pm

I understand.. I'm going to run some tests later.
Thank you

asdffdsa · October 20, 2021, 2:04pm

hi,
you can use simple regfex patterns with rclone.
for example,
this will work {\d\d\d\d\d\d\d\d}
this will not work {\d{8}}
Failed to load filters: can't nest '{' '}' in glob "*.{\\d{8}}"

fwiw, i use the following website when testing go regex.
https://regoio.herokuapp.com/

system · November 19, 2021, 2:04pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.