Lsjson ignoring filters

[kqed (Swensed Milk)]

What is the problem you are having with rclone?

I'm writing a script that has a search function. The ls command return results perfectly with filters, but lsjson does not. And I need to use it for the script as it includes metadata i can easily parse.

Isjson seems to completely ignore filters and instead lists all folders at the root of the folder path. The same arguments/filters when used with ls returns what I want.

Run the command 'rclone version' and share the full output of the command.

rclone v1.64.2

• os/version: Microsoft Windows 11 Pro 23H2 (64 bit)
• os/kernel: 10.0.22631.2715 (x86_64)
• os/type: windows
• os/arch: amd64
• go/version: go1.21.3
• go/linking: static
• go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Local/network storage.

The command you were trying to run (eg rclone copy /tmp remote:tmp)

This returns all folders at the root of nas:music/library. Not what I want.

rclone lsjson nas:music/library --filter "+ **foldername/**" --filter "- /**" --ignore-case

This returns everything in nas:music/library/foldername. What I want, but with lsjson

rclone ls nas:music/library --filter "+ **foldername/**" --filter "- /**" --ignore-case

Filters with lsjson seems to only be effective at the root of the folder path.

What am I doing wrong? Or is this not possible?

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[XXX]
type = combine
upstreams = XXX

A log from the command that you were trying to run with the -vv flag

This would list every file on my network.

[kapitainsky]

TL;DR

add --recursive --files-only to your rclone lsjson

And here an example and more detailed explanation:

# sample data
$ tree
.
├── AAA
│   └── aaaa.txt
├── BBB
│   └── bbbb.txt
└── cccc.txt

# list all objects
$ rclone ls .
        4 cccc.txt
        4 BBB/bbbb.txt
        6 AAA/aaaa.txt

# lsf behaves the same as lsjson but easier to see difference from ls
# list all objects and directories in the path
$ rclone lsf .
AAA/
BBB/
cccc.txt

# the same as lsf but json output formatting is added
$ rclone lsjson .
[
{"Path":"AAA","Name":"AAA","Size":-1,"MimeType":"inode/directory","ModTime":"2023-11-22T14:49:27.467234492Z","IsDir":true},
{"Path":"BBB","Name":"BBB","Size":-1,"MimeType":"inode/directory","ModTime":"2023-11-22T14:49:06.569503791Z","IsDir":true},
{"Path":"cccc.txt","Name":"cccc.txt","Size":4,"MimeType":"text/plain; charset=utf-8","ModTime":"2023-11-22T14:56:49.520605230Z","IsDir":false}
]


# this way I can list all files using lsf and only then apply filter
$ rclone lsf . --recursive --files-only
cccc.txt
BBB/bbbb.txt
AAA/aaaa.txt

Now when you apply --filter "+ **AAA/**" --filter "- /**" this is how rclone interprets it (you can display it by adding --dump filters flag to your command):

--- start filters ---
--- File filter rules ---
+ (^|/).*AAA/.*$
- ^.*$
--- Directory filter rules ---
+ (^|/).*AAA/.*/$
+ (^|/).*AAA/$
+ (^|/).*/$
+ (^|/).*AAA/.*$
- ^.*$
--- end filters ---

If you work it out line by line you will see that results you are getting are perfectly correct.

I think confusion originates from the fact that different ls* commands have different purpose:

ls - lists objects only in the path
lsjson (like lsf) - lists files and directories in the path

Documentation could be more clear here about rather nuanced difference between files/objects and the fact that directories are often only imaginary in cloud worlds. ls shows you real objects present at the remote. lsf and lsjson try to be more human friendly and present "files" and "directories". IMO it is only confusing but has been for so long that now it is sweet legacy which will live forever.

And you can see in my example above that even if you apply the same filter --filter "+ **AAA/**" --filter "- /**" but when it is applied against different input results (like ls vs lsjson) output will be different too.

[kqed (Swensed Milk)]

Thank you so much for this. It works perfectly now.

@kapitainsky

I think confusion originates from the fact that different ls* commands have different purpose

I forgot this and i thought they just display results differently and that's it.

This was a big help. Thank you again.