Filtering with lsf command

What is your rclone version (output from rclone version)

rclone v1.57.0

  • os/version: Microsoft Windows 10 Pro N 2009 (64 bit)
  • os/kernel: 10.0.19043.1348 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.17.2
  • go/linking: dynamic
  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The rclone config contents with secrets removed.

[mdrive]
type = drive
client_id = xxx
client_secret = yyy
service_account_file = 
token = zzz
root_folder_id = vvv

[mcrypt]
type = crypt
remote = mdrive:backup
filename_encryption = standard
directory_name_encryption = true
password = www
password2 = 

What is the problem you are having with rclone?

Let's assume we have the following directory/file structure

/h/
├── a/
│   ├── a1/
│   │   └── a1.txt
│   └── a.txt
│   
├── b/
│   ├── b1/
│   │   └── b1.txt
│   └── b.txt
│
├── c/
.   .
.   .

The structure of the subfolders a/ and b/ of /h/ shall continue in /h/ for subfolders c/, d/, ...

Command:
rclone lsf mcrypt:/h/a

Result:

a1/
a.txt

Command:
rclone lsf -R mcrypt:/h/a

Result:

a1/
a.txt
a1/a1.txt

Now I want to list several folders in /h/ with a filter. Let's say a/ and b/. I got it working recursively using
rclone lsf -R mcrypt:/h --include "a/**" --include "b/**"

a/
a/a1/
a/a.txt
a/a1/a1.txt
b/
b/b1/
b/b.txt
b/b1/b1.txt

What I want to achieve now is listing the contents of the included subfolders without recursion, displaying files and folders.
So the desired output would look like this:

a/
a/a1/
a/a.txt
b/
b/b1/
b/b.txt

I tried the following command
rclone lsf -R mcrypt:/h --include "a/*" --include "b/*"

a/
a/a.txt
b/
b/b.txt

As you can see, the output of the folders a1 and b1 is omitted. I tried some other variations of the command but was not able to achieve the desired output.
Is it possible to do it with just one command?

  • won't match the slash on a directory.
felix@gemini:~/test$ rclone lsf /home/felix/test -R
a/
b/
c/
a/a.txt
a/a1/
a/a1/a1.txt
b/b.txt
b/b1/
b/b1/b1.txt
felix@gemini:~/test$ rclone lsf /home/felix/test -R --include "a/*"
a/
a/a.txt
felix@gemini:~/test$ rclone lsf /home/felix/test -R --include "a/**"
a/
a/a.txt
a/a1/
a/a1/a1.txt

https://rclone.org/filtering/#pattern-syntax

So you need ** like I shared above.

Thanks for your reply.
However the output is not the desired one described above.

With a bit more trial and error I was now able to achieve the intended output but I don't understand the behavior.

rclone lsf -R "mcrypt:/h/a/*/"
NO OUTPUT (why)

rclone lsf -R "mcrypt:/h/a/*/ " (space after last slash)

a/
a/a1/

(again why)

So I get the desired output with the following command:
rclone lsf -R "mcrypt:/h/a/*/ " --include "mcrypt:/h/a/*"

a/
a/a1/
a/a.txt

You can't use a glob in there as it shouldn't work. Can you share full output of those commands with -vv --dump-filters?

felix@gemini:~/test$ rclone lsf DB:test -R
h/
h/a/
h/b/
h/b/b.txt
h/b/b1/
h/a/a.txt
h/a/a1/
h/b/b1/b1.txt
h/a/a1/a1.txt
felix@gemini:~/test$ rclone lsf "DB:test/h/a/*/" -R
2021/11/27 07:29:47 ERROR : : error listing: directory not found
2021/11/27 07:29:47 Failed to lsf with 2 errors: last error was: error in ListJSON: directory not found
felix@gemini:~/test$ rclone lsf "DB:test/h/a/*/ " -R
2021/11/27 07:29:51 ERROR : : error listing: directory not found
2021/11/27 07:29:51 Failed to lsf with 2 errors: last error was: error in ListJSON: directory not found

rclone filters on files and prints files out which I think is the confusion. Your output won't work as it won't ever print a/a1 like you have since no files match.

Sorry, I made a mistake in my last post and can't edit it. Here's the correct version:

With a bit more trial and error I was now able to achieve the intended output but I don't understand the behavior.

rclone lsf -R "mcrypt:/h/" --include "mcrypt:/h/a/*/"
NO OUTPUT (why)

rclone lsf -R "mcrypt:/h/" --include "mcrypt:/h/a/*/ " (space after last slash)

a/
a/a1/

(again why)

So I get the desired output with the following command:
rclone lsf -R "mcrypt:/h/" --include "mcrypt:/h/a/*/ " --include "mcrypt:/h/a/*"

a/
a/a1/
a/a.txt

Output of last command with --v --dump filters:

rclone lsf -R "mcrypt:/h/" --include "a/*/ " --include "a/*" -vv --dump filters
--- start filters ---
--- File filter rules ---
+ (^|/)a/[^/]*/ $
+ (^|/)a/[^/]*$
- ^.*$
--- Directory filter rules ---
+ (^|/)a/[^/]*/$
+ (^|/)a/$
- ^.*$
--- end filters ---
2021/11/29 02:05:45 DEBUG : rclone: Version "v1.57.0" starting with parameters ["D:\\Userdata\\bin\\rclone.exe" "lsf" "-R" "mcrypt:/h/" "--include" "a/*/ " "--include" "a/*" "-vv" "--dump" "filters"]
2021/11/29 02:05:45 DEBUG : Creating backend with remote "mcrypt:/h/"
2021/11/29 02:05:45 DEBUG : Using config file from "C:\\Users\\myname\\.config\\rclone\\rclone.conf"
2021/11/29 02:05:45 DEBUG : Creating backend with remote "mdrive:/backup/8ecjk22tnvfejgul0s1j4bfq7g"
2021/11/29 02:05:46 DEBUG : fs cache: renaming cache item "mdrive:/backup/8ecjk22tnvfejgul0s1j4bfq7g" to be canonical "mdrive:kh/8ecjk22tnvfejgul0s1j4bfq7g"
2021/11/29 02:05:46 DEBUG : fs cache: switching user supplied name "mdrive:/backup/8ecjk22tnvfejgul0s1j4bfq7g" for canonical name "mdrive:backup/8ecjk22tnvfejgul0s1j4bfq7g"
2021/11/29 02:05:46 DEBUG : b: Excluded
a/
a/a.txt
a/a1/
2021/11/29 02:05:46 DEBUG : a/a1/a1.txt: Excluded
2021/11/29 02:05:46 DEBUG : 4 go routines active

That doesn't work at all as I shared above

If you can share the debug output of what you ran, that would be great.

Same as my above comment.

That command is wrong as you have a typo there with your include.

That doesn't match the command above. Is that correct as that's the command produces the output you want?

Man, I'm so sorry to have f***ed up my post 2 times in a row.
all --include filters must not contain mcrypt:/h/.

So here's the logs:
rclone lsf -R "mcrypt:/h/" --include "a/*/" -vv --dump filters

--- start filters ---
--- File filter rules ---
- ^.*$
--- Directory filter rules ---
+ (^|/)a/[^/]*/$
- ^.*$
--- end filters ---
2021/11/29 02:02:46 DEBUG : rclone: Version "v1.57.0" starting with parameters ["D:\\Userdata\\bin\\rclone.exe" "lsf" "-R" "mcrypt:/h/" "--include" "a/*/" "-vv" "--dump" "filters"]
2021/11/29 02:02:46 DEBUG : Creating backend with remote "mcrypt:/h/"
2021/11/29 02:02:46 DEBUG : Using config file from "C:\\Users\\myname\\.config\\rclone\\rclone.conf"
2021/11/29 02:02:46 DEBUG : Creating backend with remote "mdrive:/kh/8ecjk22tnvfejgul0s1j4bfq7g"
2021/11/29 02:02:47 DEBUG : fs cache: renaming cache item "mdrive:/kh/8ecjk22tnvfejgul0s1j4bfq7g" to be canonical "mdrive:kh/8ecjk22tnvfejgul0s1j4bfq7g"
 name "mdrive:kh/8ecjk22tnvfejgul0s1j4bfq7g"
2021/11/29 02:02:47 DEBUG : b: Excluded
2021/11/29 02:02:47 DEBUG : a: Excluded
2021/11/29 02:02:47 DEBUG : 4 go routines active

rclone lsf -R "mcrypt:/h/" --include "a/*" -vv --dump filters

--- start filters ---
--- File filter rules ---
+ (^|/)a/[^/]*$
- ^.*$
--- Directory filter rules ---
+ (^|/)a/$
- ^.*$
--- end filters ---
2021/11/29 20:10:06 DEBUG : rclone: Version "v1.57.0" starting with parameters ["D:\\Userdata\\bin\\rclone.exe" "lsf" "-R" "mcrypt:/h/" "--include" "a/*" "-vv" "--dump" "filters"]
2021/11/29 20:10:06 DEBUG : Creating backend with remote "mcrypt:/h/"
2021/11/29 20:10:06 DEBUG : Using config file from "C:\\Users\\myname\\.config\\rclone\\rclone.conf"
2021/11/29 20:10:06 DEBUG : Creating backend with remote "mdrive:/kh/8ecjk22tnvfejgul0s1j4bfq7g"
2021/11/29 20:10:07 DEBUG : fs cache: renaming cache item "mdrive:/kh/8ecjk22tnvfejgul0s1j4bfq7g" to be canonical "mdrive:kh/8ecjk22tnvfejgul0s1j4bfq7g"
2021/11/29 20:10:07 DEBUG : fs cache: switching user supplied name "mdrive:/kh/8ecjk22tnvfejgul0s1j4bfq7g" for canonical name "mdrive:kh/8ecjk22tnvfejgul0s1j4bfq7g"
2021/11/29 20:10:07 DEBUG : b: Excluded
a/
2021/11/29 20:10:08 DEBUG : a/a1: Excluded
a/a.txt
2021/11/29 20:10:08 DEBUG : 4 go routines active

rclone lsf -R "mcrypt:/h/" --include "a/*/ " -vv --dump filters

--- start filters ---
--- File filter rules ---
+ (^|/)a/[^/]*/ $
- ^.*$
--- Directory filter rules ---
+ (^|/)a/[^/]*/$
+ (^|/)a/$
- ^.*$
--- end filters ---
2021/11/29 02:03:06 DEBUG : rclone: Version "v1.57.0" starting with parameters ["D:\\Userdata\\bin\\rclone.exe" "lsf" "-R" "mcrypt:/h/" "--include" "a/*/ " "-vv" "--dump" "filters"]
2021/11/29 02:03:06 DEBUG : Creating backend with remote "mcrypt:/h/"
2021/11/29 02:03:06 DEBUG : Using config file from "C:\\Users\\myname\\.config\\rclone\\rclone.conf"
2021/11/29 02:03:06 DEBUG : Creating backend with remote "mdrive:/kh/8ecjk22tnvfejgul0s1j4bfq7g"
2021/11/29 02:03:07 DEBUG : fs cache: renaming cache item "mdrive:/kh/8ecjk22tnvfejgul0s1j4bfq7g" to be canonical "mdrive:kh/8ecjk22tnvfejgul0s1j4bfq7g"
2021/11/29 02:03:07 DEBUG : fs cache: switching user supplied name "mdrive:/kh/8ecjk22tnvfejgul0s1j4bfq7g" for canonical name "mdrive:kh/8ecjk22tnvfejgul0s1j4bfq7g"
2021/11/29 02:03:08 DEBUG : b: Excluded
2021/11/29 02:03:08 DEBUG : a/a.txt: Excluded
a/a1/
2021/11/29 02:03:08 DEBUG : a/a1/a1.txt: Excluded
2021/11/29 02:03:08 DEBUG : 4 go routines active

The log for the combination rclone lsf -R "mcrypt:/h/" --include "a/*/ " --include "a/*" -vv --dump filters was correct in my last post and indeed is the output I want. I don't understaand the workings though because as you wrote above the filters with single * are supposed to apply to files only and not folders.

Sorry as I didn't quite state it well/correct. A * will match one directory and a ** will match recursively as that is what I trying to say in a better fashion.

https://rclone.org/filtering/#how-filter-rules-are-applied-to-directories

There is a nice GO Regex tester that I use as they are a little different from normal regexes.

https://regoio.herokuapp.com/

You can play around with the filters and test things as it's not quite perfect since rclone will evaluate a directory match and a file match based on the filters you can see in the debug:

--- start filters ---
--- File filter rules ---
+ (^|/)a/[^/]*/ $
+ (^|/)a/[^/]*$
- ^.*$
--- Directory filter rules ---
+ (^|/)a/[^/]*/$
+ (^|/)a/$
- ^.*$
--- end filters ---

You can see which include is matching the file and which is matching the directory.

felix@gemini:~$ rclone lsf -R 'DB:h/' --include "a/*/ " --include "a/*"
a/
a/a.txt
a/a1/
felix@gemini:~$ rclone lsf -R 'DB:h/' --include "a/*/ "
a/
a/a1/
felix@gemini:~$ rclone lsf -R 'DB:h/' --include "a/*"
a/
a/a.txt

The "**" will recurse:

 rclone lsf -R 'DB:h/' --include "a/**"
a/
a/a.txt
a/a1/
a/a1/a1.txt

I think you hit this as well as listing files using lsf produces this bug:

I am not quite sure the use case you have for listing a directory.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.