Rclone move filter + errors

Help and Support
1

What is the problem you are having with rclone?

I tried to use filter in rclone move to move the files containing "mide" in the target directory (either the files in the root directory of the target directory or the files in the subdirectories) into the mide folder, but when I run the command It is found that many files that do not contain mide have also moved past, such as some files containing only m and files containing only d, but the files that actually contain mide have not passed, such as files in subdirectories

What is your rclone version (output from rclone version)

1.52.1

Which OS you are using and how many bits (eg Windows 7, 64 bit)

debian 10

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

gclone move gc:xxx gc:yyy -vvP --filter '+ *'mide'*'

The rclone config contents with secrets removed.

Paste config here

A log from the command with the -vv flag

Paste  log here
2

You need to add:

-filter '- *'

By default it's inclusive based on your first filter so you need to exclude everything you don't want.

3

Still not work, I can’t use the filter command and changed it to include. Do you think it is changed this way? In the target directory and in the subfolders under the target directory, no matter the case, as long as the mide is included, it will move over:
gclone move oneking:{1B_8ggpvu7UEvIVJbm9O3pyzCFRka697N} oneking:{1QtI6xnG-zRvQPX-wC2fgP_PZSSHGKtz8} -vvP --fast-list --include "mide.*" --ignore-case

4

You can use rclone ls to test with. Please share the outputs of your commands with -vv and I can help out.

felix@gemini:~/test$ rclone ls /home/felix/test --filter '+ *mide*' -vv
2020/06/15 08:19:23 DEBUG : rclone: Version "v1.52.1" starting with parameters ["rclone" "ls" "/home/felix/test" "--filter" "+ *mide*" "-vv"]
2020/06/15 08:19:23 DEBUG : Using config file from "/opt/rclone/rclone.conf"
        0 test1
        0 test4
        0 testmide2
        0 testmide3
2020/06/15 08:19:23 DEBUG : 2 go routines active
felix@gemini:~/test$ rclone ls /home/felix/test --filter '+ *mide*' -vv --filter '- *'
2020/06/15 08:19:30 DEBUG : rclone: Version "v1.52.1" starting with parameters ["rclone" "ls" "/home/felix/test" "--filter" "+ *mide*" "-vv" "--filter" "- *"]
2020/06/15 08:19:30 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2020/06/15 08:19:30 DEBUG : test1: Excluded
2020/06/15 08:19:30 DEBUG : test4: Excluded
        0 testmide2
        0 testmide3
2020/06/15 08:19:30 DEBUG : 2 go routines active
6

Are you trying to match mide and MIDE?

Is there a line that you want matched in that output?

7

Yes, both mide and MIDE match, but as you can see, the results outside mide are also

8

If you want to. match a directory and all of its contents:

felix@gemini:~/test$ rclone ls /home/felix/test --filter '+ *mide*' -vv --filter '- *'
2020/06/15 08:25:59 DEBUG : rclone: Version "v1.52.1" starting with parameters ["rclone" "ls" "/home/felix/test" "--filter" "+ *mide*" "-vv" "--filter" "- *"]
2020/06/15 08:25:59 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2020/06/15 08:25:59 DEBUG : test1: Excluded
2020/06/15 08:25:59 DEBUG : test4: Excluded
        0 testmide2
        0 testmide3
2020/06/15 08:25:59 DEBUG : testmide14/hosts: Excluded
2020/06/15 08:25:59 DEBUG : testmide14/test: Excluded
        0 testmide14/testmideagain
2020/06/15 08:25:59 DEBUG : 2 go routines active
felix@gemini:~/test$ rclone ls /home/felix/test --filter '+ **mide**' -vv --filter '- *'
2020/06/15 08:28:34 DEBUG : rclone: Version "v1.52.1" starting with parameters ["rclone" "ls" "/home/felix/test" "--filter" "+ **mide**" "-vv" "--filter" "- *"]
2020/06/15 08:28:34 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2020/06/15 08:28:34 DEBUG : test1: Excluded
2020/06/15 08:28:34 DEBUG : test4: Excluded
        0 testmide2
        0 testmide3
        0 testmide14/hosts
        0 testmide14/test
        0 testmide14/testmideagain
2020/06/15 08:28:34 DEBUG : 2 go routines active

You use **

9

No, MIDE does not match on your filters. The filter only match files with mide and they have to contain mide in them.

10

Thanks for your reply, I am running the include command, if the include can meet the requirements, there is no need to filter, it is too cumbersome

11

There are always many ways to do things so use what works best.

I find filtering to be easier as I can tailor the rules much more specific.

I tend to use files-from and build my logic of what I need as it's top down processing and work for my brain.

If the goal is to filter all files under any directory that contains "mide" in the name, it's just

+ **mide**/**
- *

If it's mide and MIDE

+ **mide**/**
+ "**MIDE**/**"
- *

And as always test with dry-run and with ls and you can validate your commands do what you'd expect. I use rlcone ls first and graduate to dry-run on the copy, sync or move command and once dry-run is validated, I use the actual command.
12

mide Mide mIde miDe miDE,MIDE...........Etc.,
"--ignore-case" is invalid for "filter"?

13

You can use ignore case.

felix@gemini:~/test$ rclone ls /home/felix/test --filter '+ **mide**' -vv --filter '- *' --ignore-case
2020/06/15 08:56:46 DEBUG : rclone: Version "v1.52.1" starting with parameters ["rclone" "ls" "/home/felix/test" "--filter" "+ **mide**" "-vv" "--filter" "- *" "--ignore-case"]
2020/06/15 08:56:46 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2020/06/15 08:56:46 DEBUG : test: Excluded
        0 testMIde23
        0 testmide124
2020/06/15 08:56:46 DEBUG : 2 go routines active
14

There is a problem, that is, the files in the sub-files are not moved separately, but moved together with the sub-folders. How to make the move command only move the files that meet the conditions, but not the folders that meet the conditions

15

Do you want to flatten the structure? I'm trying to make sure I undersatnd.

Can you share an example of before and after with a few file example?

17

That doesn't exist in rclone currently.

You want flatten and this is the issue for it:

18

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.