Exclude & include & --filter-from problem

Help and Support
1

What is the problem you are having with rclone?

I am trying to move Folders Alphabetically, but skip certain file extensions inside the folders.

Tried several variants and even read:

I am sure it is my syntax but cannot figure out what I am doing wrong.

The goal is to got into a remote:folder, copy all folders inside there via
--include "/[aA]**"

But exclude specific file extensions.

I thought by reading issue 287 the
--exclude=*.{jpg,png,gif} --include "/[aA]**"

Would work, but it does not, and even the --filter-from is still moving the file extensions I do not want moved.

What is I am doing wrong?

rclone v1.61.1

  • os/version: Microsoft Windows 10 Enterprise for Virtual Desktops 22H2 (64 bit)
  • os/kernel: 10.0.19045.2364 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.19.4
  • go/linking: static
  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

**1st Command**
rclone -vP --fast-list --delete-empty-src-dirs --transfers=20 --exclude=*.{jpg,png,gif} --include "/[aA]**" move BOT-11:"/00.Folders/" BOT-A:/00/ --config=drives.conf --dry-run


**2nd Command**
rclone -vP --fast-list --delete-empty-src-dirs --transfers=20 --include "/[aA]**" --filter-from excludefiles.txt move BOT--11:"/00.Folders/" BOT-A:/00/ --config=drives.conf

**excludefiles.txt CONTENTS**

- *.jpg
- *.png
- *.gif


**3rd Command**
rclone -vP --fast-list --delete-empty-src-dirs --transfers=20 --filter-from excludefiles.txt move BOT-11:"/00.Folders/" BOT-A:/00/ --config=drives.conf

**excludefiles.txt CONTENTS**

+ "/[aA]**"
- *.jpg
- *.png
- *.gif

The rclone config contents with secrets removed.

[BOT-11]
type = drive
scope = drive
service_account_file = sa\5000.json
team_drive = 
server_side_across_configs = true

[BOT-A]
type = drive
scope = drive
service_account_file = sa\5001.json
team_drive = 
server_side_across_configs = true

A log from the command with the -vv flag

Dry Run does not help here, and running command far too many folders/files etc
2

That feels a bit more complex than it needs to be and mixing include and exclude rules can get a bit wonky.

I'd use filter from and it's easier and rules are processed top down.

I think this might be what you want.

[felix@gemini test]$ rclone ls /home/felix/test --filter-from /home/felix/filter
        0 a/blah
        0 A/copyme
[felix@gemini test]$ rclone ls /home/felix/test
        0 one/blah
        0 b/testing
        0 a/blah
        0 a/blah.jpg
        0 a/blah.png
        0 A/copyme
        0 A/three.png
        0 A/two.jpg
[felix@gemini test]$ cat ../filter
- *.jpg
- *.png
- *.gif
+ [aA]/**
- **

filter-from is top down and it will stop on a match. I left it with a few more rules to be clear as you can combine the first 3 excludes.

3

Putting this in the file

  • [aA]/**
  • **

Does not work properly for me. That said if I do this

  • /[aA]**
  • **

it only shows me the folders starting with aA, which is what I want.

But when I run it as a move command, it iterates all directories even those Directories not starting with aA

Why is the move command iterating directories that do NOT start with aA?

Is there a way to make sure it only looks at Directories starting with the letters that have been chosen?

Asking because there are 100,000 folders inside this directory, so right now it means rclone is going to iterate these directories 360,000 times if I try and move directories starting with A-Z and 0-9, that is a lot of calls and even time for it to process.

Even using SA accounts causes the pacer to appear and slow it down

Honestly never noticed this before, as I seldom run -vv just -v

Also I noted that during transfers it would ignore *.jpg but it moves *.JPG so now I am re running it with

--ignore-case

adding this has appeared to fix the issue where it now ignores upper case extensions

4

Sorry as I have no idea what you put in or what you are doing/running.

You'd have to share your actual filter file in total and what command you are running.

5

ok running this command

rclone -vv --filter-from excludefiles.txt ls BOT-11:"/00.Folders/" --config=drives.conf


with this inside excludefiles.txt


- *.jpg
- *.jpeg
- *.png
+ /[bB]**
- **

The above only shows me folders starting with bB and what contents are in it, it tells me if something is excluded.

Which is perfect! But if I run in the move command totally different out put. This is my move command

rclone -vvP --fast-list --ignore-case --delete-empty-src-dirs --transfers=20 --filter-from excludefiles.txt move BOT-11:"/00.Folders/" BOT-B:/00/ --config=drives.conf

with this inside excludefiles.txt


- *.jpg
- *.jpeg
- *.png
+ /[bB]**
- **

The above move command iterates every folder and every file inside those folders, even the ones not starting with the letter bB

Do not understand why ls skips directories not starting with bB but move shows everything.

I had assumed that + /[bB]** flag would prevent rclone move from looking inside folders not starting with bB.

Not sure what I am missing or doing wrong.

6

You have to enclose things in code blocks or it'll auto format and I have no idea what is in that file.

You want to put three backtics `

before and after a section

like

this

and it'll make a code block. Put your filter file in a code block as I can't tell if you are doing what I shared or not.

7

fixed sorry about that

8

Looks good.

Can you share a snippet of the rclone ls and share the output of what is not working with the move?

9

It is traversing every directory and sub directory inside the root folder.

What I am trying to do is only move directories starting with bB in the root, and skip certain files inside those folders from being moved.

But rclone is traversing every directory to its entire depth.

Should I be adding the --no-traverse to the options? Will using that command prevent the excluded extensions form being looked at?

2023-01-05 17:19:06 DEBUG : SaleWebDesign.Com-Excel-VBA-Macros-Advanced-Course/SaleWebDesign.Comf.jpg: Excluded from sync (and deletion)
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:        24.1s
2023-01-05 17:19:06 DEBUG : SaleWebDesign.Com-Excel-VBA-Macros-Advanced-Course/[SaleWebDesign.Com].txt: Excluded from sync (and deletion)
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:        24.1s
2023-01-05 17:19:06 DEBUG : SaleWebDesign.Com-Excel-VBA-Macros-Advanced-Course/Read Me.txt: Excluded from sync (and deletion)
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:        24.1s
2023-01-05 17:19:06 DEBUG : SaleWebDesign.Com-Excel-VBA-Macros-Advanced-Course/Download More Free Coures.txt: Excluded from sync (and deletion)
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:        24.1s
2023-01-05 17:19:06 DEBUG : SaleWebDesign.Com-Excel-VBA-Macros-Advanced-Course/SaleWebDesign.Com.png: Excluded from sync (and deletion)
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:        24.1s
2023-01-05 17:19:06 DEBUG : SaleWebDesign.Com-Excel-VBA-Macros-Advanced-Course/GET 100% OFF COUPONS.txt: Excluded from sync (and deletion)
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:        24.1s
2023-01-05 17:19:06 DEBUG : SaleWebDesign.Com-Excel-VBA-Macros-Advanced-Course/06 - Msgbox and Inputbox: Excluded from sync (and deletion)
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:        24.1s
2023-01-05 17:19:06 DEBUG : SaleWebDesign.Com-Excel-VBA-Macros-Advanced-Course/04 - Operators: Excluded from sync (and deletion)
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:        24.1s
2023-01-05 17:19:06 DEBUG : SaleWebDesign.Com-Excel-VBA-Macros-Advanced-Course/10 - String Functions in Excel VBA: Excluded from sync (and deletion)
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:        24.1s
2023-01-05 17:19:06 DEBUG : SaleWebDesign.Com-Excel-VBA-Macros-Advanced-Course/08 - Conditional Statements: Excluded from sync (and deletion)
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:        24.1s
2023-01-05 17:19:06 DEBUG : SaleWebDesign.Com-Excel-VBA-Macros-Advanced-Course/02 - Introduction to VBA: Excluded from sync (and deletion)
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:        24.1s
2023-01-05 17:19:06 DEBUG : SaleWebDesign.Com-Excel-VBA-Macros-Advanced-Course/05 - Functions and Sub-Procedures: Excluded from sync (and deletion)
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:        24.1s
2023-01-05 17:19:06 DEBUG : SaleWebDesign.Com-Excel-VBA-Macros-Advanced-Course/07 - Excel Object Model: Excluded from sync (and deletion)
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:        24.1s
2023-01-05 17:19:06 DEBUG : SaleWebDesign.Com-Excel-VBA-Macros-Advanced-Course/09 - Loops: Excluded from sync (and deletion)
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:        24.1s
2023-01-05 17:19:06 DEBUG : SaleWebDesign.Com-Excel-VBA-Macros-Advanced-Course/03 - Data Types and Variables: Excluded from sync (and deletion)
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:        24.1s
2023-01-05 17:19:06 DEBUG : SaleWebDesign.Com-Excel-VBA-Macros-Advanced-Course/01 - Course Introduction: Excluded from sync (and deletion)
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:        24.1s
2023-01-05 17:19:06 DEBUG : SaleWebDesign.Com-Excel-VBA-Macros-Advanced-Course/12 - Events and Userforms: Excluded from sync (and deletion)
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:        24.1s
2023-01-05 17:19:06 DEBUG : SaleWebDesign.Com-Excel-VBA-Macros-Advanced-Course/11 - Date Functions in Excel VBA: Excluded from sync (and deletion)
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:        24.1s
2023-01-05 17:19:06 DEBUG : SaleWebDesign.Com-Excel-VBA-Macros-Advanced-Course/16 - Thank you: Excluded from sync (and deletion)
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:        24.1s
2023-01-05 17:19:06 DEBUG : SaleWebDesign.Com-Excel-VBA-Macros-Advanced-Course/15 - VBA Tips: Excluded from sync (and deletion)
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:        24.1s
2023-01-05 17:19:06 DEBUG : SaleWebDesign.Com-Excel-VBA-Macros-Advanced-Course/14 - Practical Examples: Excluded from sync (and deletion)
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:        24.1s
2023-01-05 17:19:06 DEBUG : SaleWebDesign.Com-Excel-VBA-Macros-Advanced-Course/13 - Error Handling: Excluded from sync (and deletion)
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:        24.1s

2023-01-05 17:19:06 DEBUG : CA INTER_IPCC Income Tax Classes [PLSRCf1nrcNgkMJTpB1T4g_Dm2maPTKO1a].NA/Introduction of Income Tax Course & Its Journey _ By CA Vikas Sharma.f251.webm: Excluded from sync (and deletion)
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:        24.1s
2023-01-05 17:19:06 DEBUG : CA INTER_IPCC Income Tax Classes [PLSRCf1nrcNgkMJTpB1T4g_Dm2maPTKO1a].NA/CA Inter_IPCC Income Tax Basic Introduction Paper 4 _ Class Part C.f248.webm.part-Frag1.part: Excluded from sync (and deletion)
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:        24.1s
2023-01-05 17:19:06 DEBUG : CA INTER_IPCC Income Tax Classes [PLSRCf1nrcNgkMJTpB1T4g_Dm2maPTKO1a].NA/'Exam Strategy from Ravana' रावण से सीखे एग्जाम की तैयारी _ CA Rahul Jain.f247.webm.part-Frag1.part: Excluded from sync (and deletion)
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:        24.1s
2023-01-05 17:19:06 DEBUG : CA INTER_IPCC Income Tax Classes [PLSRCf1nrcNgkMJTpB1T4g_Dm2maPTKO1a].NA/Live Session 3 For CA INTER _ IPCC Students Paper 4 Taxation With MCQ Quiz.f247.webm.part: Excluded from sync (and deletion)
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:        24.1s
2023-01-05 17:19:06 DEBUG : CA INTER_IPCC Income Tax Classes [PLSRCf1nrcNgkMJTpB1T4g_Dm2maPTKO1a].NA/CA Inter_IPCC Income Tax Basic Introduction Paper 4 _ Class Part A.f248.webm.ytdl: Excluded from sync (and deletion)
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:        24.1s

2023-01-05 17:19:07 DEBUG : React - The Complete Guide (incl Hooks, React Router, Redux) /24 Course Roundup: Excluded from sync (and deletion)
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:        25.0s
2023-01-05 17:19:07 DEBUG : React - The Complete Guide (incl Hooks, React Router, Redux) /10 Advanced_ Handling Side Effects, Using Reducers & Using the Context API: Excluded from sync (and deletion)
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:        25.0s
10

Even using the --no-traverse flag rclone is still checking every directory whether it starts with the letter or not, and checking every sub directory and file.

So trying to move 100,000 folder from the source to Alphabetical Destination means I will index 100,000 folders 360,000 times. The way it is doing it now that is.

11

This is the cause --fast-list - lose that and rclone will behave :slight_smile:

12

Holy crap, I always thought fast-list was my friend! But you are 100% right is doing it the way I expected now.

Honestly never would have thought fast-list was the issue.

Thank You, very much!

13

Ah just an off topic question.

Is there any other time I should NOT be using --fast-list? I pretty much have it built into every command I use

14

It would depend.

If you want to grab the whole list before doing something, you'd want to use it.

If you don't want to grab the whole list, don't use it.

Depending on what your goal is, you'd have to decide if you want it on or off.

15

--fast-list lists first and filters afterwards.

This means that it can't skip directories to list.

Depending on your source --fast-list may be more or less helpful.

If you don't see "ListR": true in rclone backend features remote: there is no point using it ever

$ rclone backend features s3: | grep ListR
		"ListR": true,
1 Like
16

Thank you great to know, need to go through my list of commands and see where I might be miss using it

1 Like
17

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.