Regular expressions do not work

What is the problem you regular expressionsare having with rclone?

I want to delete all files that do not end in numbers, but regular expressions do not work

Run the command 'rclone version' and share the full output of the command.

rclone v1.63.1

  • os/version: Microsoft Windows 11 Pro 22H2 (64 bit)
  • os/kernel: 10.0.22621.2134 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.20.6
  • go/linking: static
  • go/tags: cmount

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Paste command here
```rclone delete D:\yy  --exclude *.\d+    --progress -vv


#### The rclone config contents with secrets removed.  
<!--  You should use 3 backticks to begin and end your paste to make it readable.   -->

Paste config here




#### A log from the command that you were trying to run with the `-vv` flag  
<!-- You should use 3 backticks to begin and end your paste to make it readable.  Or use a service such as https://pastebin.com or https://gist.github.com/   -->

Paste log here

2023/09/04 02:47:03 DEBUG : Creating backend with remote "D:\\yy"
2023/09/04 02:47:03 DEBUG : Using config file from "C:\\Users\\11971\\AppData\\Roaming\\rclone\\rclone.conf"
2023/09/04 02:47:03 DEBUG : fs cache: renaming cache item "D:\\yy" to be canonical "//?/D:/yy"
2023-09-04 02:47:03 DEBUG : Waiting for deletions to finish
2023-09-04 02:47:03 INFO  : IMG_20211118_035338_635.jpg.1: Deleted
2023-09-04 02:47:03 INFO  : IMG_20211118_035317_881.jpg.1: Deleted
2023-09-04 02:47:03 INFO  : IMG_20211114_131136_864.jpg.1: Deleted
2023-09-04 02:47:03 INFO  : IMG_20211114_064125_214.jpg.1: Deleted
2023-09-04 02:47:03 INFO  : IMG_20211118_032106_982.jpg.1: Deleted
2023-09-04 02:47:03 INFO  : IMG_20211114_131112_328.jpg.1: Deleted
2023-09-04 02:47:03 INFO  : IMG_20211118_035322_213.jpg.1: Deleted
2023-09-04 02:47:03 INFO  : IMG_20211114_131127_257.jpg.1: Deleted
2023-09-04 02:47:03 INFO  : IMG_20211118_131733_920.jpg.1: Deleted
2023-09-04 02:47:03 INFO  : IMG_20211118_131742_882.jpg.1: Deleted
2023-09-04 02:47:03 INFO  : IMG_20211122_154651_340.jpg.1: Deleted
2023-09-04 02:47:03 INFO  : IMG_20211122_154738_095.jpg.1: Deleted
2023-09-04 02:47:03 INFO  : IMG_20211122_154810_725.jpg.1: Deleted
2023-09-04 02:47:03 INFO  : IMG_20211122_154813_641.jpg.1_f21ap2: Deleted
2023-09-04 02:47:03 INFO  : IMG_20211122_154831_511.jpg.1_f21bhb: Deleted
2023-09-04 02:47:03 INFO  : IMG_20211122_154928_355.jpg.1_f21d4d: Deleted
2023-09-04 02:47:03 INFO  : IMG_20211122_154944_895.jpg.1_f21c8f: Deleted

as per rclone docs
Regular expressions should be enclosed in {{ }}

fwiw, given that regex apply only to source only:

  • might be safer to test rclone ls
    if you must use rclone delete add --dry-run
  • use -vv --dump filters to see what filters rclone is using

There are also many online tools you can use to play and experiment with regex.

My favourite is:

PS C:\Users\11971> rclone delete D:\yy --exclude *.{{\d+}} --progress -vv
Error: unknown shorthand flag: 'e' in -encodedCommand
Usage:
rclone delete remote:path [flags]

Flags:
-h, --help help for delete
--rmdirs rmdirs removes empty directories but leaves root intact

Use "rclone [command] --help" for more information about a command.
Use "rclone help flags" for to see the global flags.
Use "rclone help backends" for a list of supported services.

2023/09/04 03:06:56 Fatal error: unknown shorthand flag: 'e' in -encodedCommand

well, your regex is not valid and not sure what you are trying to do.
as mentioned, to test regex

  1. goto https://regex101.com/
  2. on the left column, click Golang
  3. type the regex

in your case
image

And very important is to understand how rclone {{}} transformation works:

{{regexp}}  becomes (^|/)(regexp)$

you can always see how your regex is "understood" by adding --dump filters to your command:

$rclone ls test: --exclude {{}} --dump filters
2023/09/03 20:16:30 NOTICE: Automatically setting -vv as --dump is enabled
--- start filters ---
--- File filter rules ---
- (^|/)()$
--- Directory filter rules ---
--- end filters ---

I want to delete all files in the folder that do not end in numbers. How do I write the delete command

*.{{\d+}} Is there any error in the expression

PS C:\Users\11971> rclone ls D:\yy --exclude *.{{\d+}} --dump filters
Error: unknown shorthand flag: 'e' in -encodedCommand
Usage:
rclone ls remote:path [flags]

Flags:
-h, --help help for ls

Use "rclone [command] --help" for more information about a command.
Use "rclone help flags" for to see the global flags.
Use "rclone help backends" for a list of supported services.

2023/09/04 03:30:47 Fatal error: unknown shorthand flag: 'e' in -encodedCommand

Sorry, I still don't quite understand

I am not Windows user but most likely *.{{\d+}} requires some characters to be escaped

maybe this, but onus on you to test,test,test

rclone ls d:\files\regextest 
        0 IMG_20211118_035338_635.jpg
        0 IMG_20211118_035338_635.jpg.1

rclone delete d:\files\regextest --exclude={{.*\d}} --dry-run 
NOTICE: IMG_20211118_035338_635.jpg: Skipped delete as --dry-run is set (size 0)

root@armbian:~# rclone delete qq:/D/yy --exclude={{.*\d}} --dry-run
2023/09/04 04:12:52 NOTICE: IMG_20211118_032106_982.jpg.1: Skipped delete as --dry-run is set (size 190.986Ki)
2023/09/04 04:12:52 NOTICE: IMG_20211118_131733_920.jpg.1: Skipped delete as --dry-run is set (size 220.571Ki)
2023/09/04 04:12:52 NOTICE: IMG_20211114_131136_864.jpg.1: Skipped delete as --dry-run is set (size 352.471Ki)
2023/09/04 04:12:52 NOTICE: IMG_20211118_131742_882.jpg.1: Skipped delete as --dry-run is set (size 285.012Ki)
2023/09/04 04:12:52 NOTICE: IMG_20211114_131127_257.jpg.1: Skipped delete as --dry-run is set (size 340.420Ki)
2023/09/04 04:12:52 NOTICE: IMG_20211122_154651_340.jpg.1: Skipped delete as --dry-run is set (size 263.725Ki)
2023/09/04 04:12:52 NOTICE: IMG_20211122_154738_095.jpg.1: Skipped delete as --dry-run is set (size 112.298Ki)
2023/09/04 04:12:52 NOTICE: IMG_20211114_064125_214.jpg.1: Skipped delete as --dry-run is set (size 298.680Ki)
2023/09/04 04:12:52 NOTICE: IMG_20211118_035317_881.jpg.1: Skipped delete as --dry-run is set (size 172.705Ki)
2023/09/04 04:12:52 NOTICE: IMG_20211118_035322_213.jpg.1: Skipped delete as --dry-run is set (size 355.070Ki)
2023/09/04 04:12:52 NOTICE: IMG_20211122_154813_641.jpg.1_f21ap2: Skipped delete as --dry-run is set (size 260.516Ki)
2023/09/04 04:12:52 NOTICE: IMG_20211122_154810_725.jpg.1: Skipped delete as --dry-run is set (size 213.568Ki)
2023/09/04 04:12:52 NOTICE: IMG_20211122_154831_511.jpg.1_f21bhb: Skipped delete as --dry-run is set (size 253.043Ki)
2023/09/04 04:12:52 NOTICE: IMG_20211122_154944_895.jpg.1_f21c8f: Skipped delete as --dry-run is set (size 368.526Ki)
2023/09/04 04:12:52 NOTICE: IMG_20211118_035338_635.jpg.1: Skipped delete as --dry-run is set (size 170.421Ki)
2023/09/04 04:12:52 NOTICE: IMG_20211114_131112_328.jpg.1: Skipped delete as --dry-run is set (size 230.674Ki)

root@armbian:~# rclone delete qq:/D/yy --exclude={{.*\d}} --progress
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Checks: 16 / 16, 100%
Deleted: 16 (files), 0 (dirs)
Elapsed time: 0.7s

I tried using another host, the system is Armbian 23.08.0 run Bullseeye with Linux 6.1.42 ophub

rclone version
rclone v1.63.1

  • os/version: debian 11.7 (64 bit)
  • os/kernel: 6.1.42-ophub (aarch64)
  • os/type: linux
  • os/arch: arm64 (ARMv8 compatible)
  • go/version: go1.20.6
  • go/linking: static
  • go/tags: none

It's strange that when I tested with dry run, the log displayed normal, but it failed when I used it

rclone touch IMG_20211118_035338_635.jpg
rclone touch IMG_20211118_035338_635.jpg.1
rclone ls .
        0 IMG_20211118_035338_635.jpg
        0 IMG_20211118_035338_635.jpg.1
rclone ls . --exclude="{{.*\d}}"
        0 IMG_20211118_035338_635.jpg

root@armbian:~# rclone delete qq:/D/yy --exclude="{{.*\d}}" --progress --dry-run --dump filters
2023/09/04 19:46:10 NOTICE: Automatically setting -vv as --dump is enabled
--- start filters ---
--- File filter rules ---

  • (^|/)(.\d)$
    --- Directory filter rules ---
    --- end filters ---
    2023/09/04 19:46:10 DEBUG : rclone: Version "v1.63.1" starting with parameters ["rclone" "delete" "qq:/D/yy" "--exclude={{.
    \d}}" "--progress" "--dry-run" "--dump" "filters"]
    2023/09/04 19:46:10 DEBUG : Creating backend with remote "qq:/D/yy"
    2023/09/04 19:46:10 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
    2023/09/04 19:46:10 DEBUG : found headers:
    2023/09/04 19:46:11 DEBUG : fs cache: renaming cache item "qq:/D/yy" to be canonical "qq:D/yy"
    2023-09-04 19:46:11 DEBUG : Waiting for deletions to finish
    2023-09-04 19:46:11 DEBUG : IMG_20211114_064125_214.jpg.1: Excluded
    2023-09-04 19:46:11 DEBUG : IMG_20211114_131112_328.jpg.1: Excluded
    2023-09-04 19:46:11 DEBUG : IMG_20211114_131127_257.jpg.1: Excluded
    2023-09-04 19:46:11 DEBUG : IMG_20211114_131136_864.jpg.1: Excluded
    2023-09-04 19:46:11 DEBUG : IMG_20211118_032106_982.jpg.1: Excluded
    2023-09-04 19:46:11 DEBUG : IMG_20211118_035317_881.jpg.1: Excluded
    2023-09-04 19:46:11 DEBUG : IMG_20211118_035322_213.jpg.1: Excluded
    2023-09-04 19:46:11 DEBUG : IMG_20211118_035338_635.jpg.1: Excluded
    2023-09-04 19:46:11 DEBUG : IMG_20211118_131733_920.jpg.1: Excluded
    2023-09-04 19:46:11 DEBUG : IMG_20211118_131742_882.jpg.1: Excluded
    2023-09-04 19:46:11 DEBUG : IMG_20211122_154651_340.jpg.1: Excluded
    2023-09-04 19:46:11 DEBUG : IMG_20211122_154738_095.jpg.1: Excluded
    2023-09-04 19:46:11 DEBUG : IMG_20211122_154810_725.jpg.1: Excluded
    2023-09-04 19:46:11 DEBUG : IMG_20211122_154813_641.jpg.1_f21ap2: Excluded
    2023-09-04 19:46:11 NOTICE: IMG_20211122_154831_511.jpg.1_f21bhb: Skipped delete as --dry-run is set (size 253.043Ki)
    2023-09-04 19:46:11 NOTICE: IMG_20211122_154928_355.jpg.1_f21d4d: Skipped delete as --dry-run is set (size 370.120Ki)
    2023-09-04 19:46:11 NOTICE: IMG_20211122_154944_895.jpg.1_f21c8f: Skipped delete as --dry-run is set (size 368.526Ki)
    Transferred: 0 B / 0 B, -, 0 B/s, ETA -
    Checks: 3 / 3, 100%
    Deleted: 3 (files), 0 (dirs)
    Elapsed time: 0.5s
    2023/09/04 19:46:11 DEBUG : 6 go routines activ

This works, but there is an exception to one file
2023-09-04 19:46:11 DEBUG : IMG_20211122_154813_641.jpg.1_f21ap2: Excluded
i try --exclude="{{.\d+$}}" ,This expression is incorrect
I hope the suffix name only has numbers. Can you make some changes? Thank you very much

If you struggle with regular expressions you can try more manual way.

  1. list all files to a file
  2. use program you are comfortable with to delete lines matching your criteria (can be for example Excel)
  3. run rclone delete --files-from file

This is an invalid block generated by chunk remote , and I want to delete them. The name of the block is *.#,so I want to delete all files with non numeric suffixes

Here you are:

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.