Copy folder content except a few sub-folders

faxotherapy · May 30, 2019, 12:11pm

Hi,
I'd like to sync the whole content of folder A except two folders which reside inside it. So, I create this filter:

- /Folder\ A/{Folder\ B,Folder\ C}/
+ /Folder\ A/**
- *

So, this should copy the whole Folder A except Folders B and C.

Unfortunately, rclone still copies everything.

Note: I read the doc.

chiraag · May 30, 2019, 12:43pm

The following worked for me and should hopefully help:

$ tree test/
test/
├── test1
│   └── a.txt
├── test2
│   └── b.txt
└── test3
    └── c.txt

$ rclone sync --dry-run --verbose --fast-list -P --filter='- /{test1,test2}/' test/ B2:mybucket/test/
2019-05-30 08:40:00 INFO  : B2 bucket mybucket path test/: Waiting for checks to finish
2019-05-30 08:40:00 INFO  : B2 bucket mybucket path test/: Waiting for transfers to finish
2019-05-30 08:40:00 NOTICE: test3/c.txt: Not copying as --dry-run
2019-05-30 08:40:00 INFO  : Waiting for deletions to finish
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 0
Checks:                 0 / 0, -
Transferred:            1 / 1, 100%
Elapsed time:        1.4s
2019/05/30 08:40:00 INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 0
Checks:                 0 / 0, -
Transferred:            1 / 1, 100%
Elapsed time:        1.4s

As you can see, only test3/c.txt was going to be transferred, as desired.

faxotherapy · May 30, 2019, 1:10pm

Thanks for your reply. I found the issue. My filter is correct, but ended up to behave weirdly because of another rule that I set before. The complete set was:

+ /\.ssh/{config,vault/keys/**}

- /Folder\ A/{Folder\ B,Folder\ C}/
+ /Folder\ A/**
- *

Rule + /\.ssh/{config,vault/keys/**} does not work and completely breaks the filtering process. Instead, I had to do the following:

+ /\.ssh/config
+ /\.ssh/vault/keys/**

- /Folder\ A/{Folder\ B,Folder\ C}/
+ /Folder\ A/**
- *

If someone can explain why doing this way doesn't work, I'd be glad to know.

+ /\.ssh/{config,vault/keys/**}

ncw · June 3, 2019, 12:30pm

If you use --dump filters you can see what regexp rclone builds. These look OK to me though...

--- File filter rules ---
+ ^\.ssh/(config|vault/keys/.*)$
- ^.*$
--- Directory filter rules ---
+ ^.*$
+ ^\.ssh/(config|vault/keys/.*)$
- ^.*$

ncw · June 4, 2019, 4:24pm

Can you make a reproduction of that? So make some files in a temporary directory and use rclone lsf -R --filter... to show the filtering going wrong?

If so then please make a new issue on github and we can fix it - thanks.

faxotherapy · June 5, 2019, 10:26pm

Test 1: OK

Test filter:

$ cat /usr/local/etc/rclone/filter_test.txt
- /**/.DS_Store

+ /\.ssh/config
+ /\.ssh/vault/keys/**

+ /\.config/mc/**
+ /\.lftp/**

- *

Applying the filter:

$ rclone lsf -R --filter-from /usr/local/etc/rclone/filter_test.txt Safe:/MBP
.config/
.lftp/
.ssh/
.ssh/config
.ssh/vault/
.ssh/vault/keys/
.ssh/vault/keys/domus_rsa.pub
.config/mc/
.lftp/cwd_history
.lftp/rc
.lftp/rl_history
.lftp/transfer_log
.config/mc/hotlist
.config/mc/hotlist.bak
.config/mc/ini
.config/mc/menu
.config/mc/panels.ini
.config/mc/panels.ini.org

Test 2: Fail

Test filter:

$ cat /usr/local/etc/rclone/filter_test.txt
- /**/.DS_Store

+ /\.ssh/{config,vault/keys/**}

+ /\.config/mc/**
+ /\.lftp/**

- *

Applying the filter:
Result: list absolutely everything that is inside MBP. Filter simply ignored.

I'm conducting further tests in an attempt to isolate the problem and make sure that if there is any issue to be raised it's a real one. I'll get back to you a bit later.

faxotherapy · June 6, 2019, 9:06am

In order to replicate the issue, I copied my .ssh folder tree while only keeping two files.

$ mkdir -p ~/ssh/vault/keys
$ cp ~/.ssh/config ~/ssh/
$ cp ~/.ssh/vault/keys/domus_rsa.pub ~/ssh/vault/keys/
$ tree ~/ssh
/Users/N/ssh
├── config
└── vault
    └── keys
        └── domus_rsa.pub

2 directories, 2 files

I create filter_test.txt:

+ /\.gitconfig
+ /\.vimrc
+ /\.zshrc
+ /\.tmux.conf

- /**/.DS_Store

+ /ssh/config
+ /ssh/vault/keys/**

- *

I put in practice filter_test.txt:

$ rclone --filter-from /usr/local/etc/rclone/filter_test.txt sync ~ Safe:/TEST/

I visit folder TEST content:

$ rclone ls Safe:TEST
      387 .gitconfig
     4906 .tmux.conf
     1930 .vimrc
    32405 .zshrc
     2105 ssh/config
      736 ssh/vault/keys/domus_rsa.pub

Now, I create filter_test_2.txt:

- /**/.DS_Store

+ /ssh/{config,vault/keys/**}

- *

I put filter_test_2.txt in practice:

$ rclone lsf -R --filter-from /usr/local/etc/rclone/filter_test_2.txt Safe:/TEST
ssh/
ssh/config
ssh/vault/
ssh/vault/keys/
ssh/vault/keys/domus_rsa.pub

And, it now works! and I don't understand why. It should have listed everything inside folder TEST.

Let's do something else. Applying filter_test_2.txt to my actual backup folder:

rclone lsf -R --filter-from /usr/local/etc/rclone/filter_test_2.txt Safe:/MBP

Bingo! Complete mess again. The filter is ignored again and all my MBP directory is listed!

So, I don't understand why applying filter_test_2.txt to folder TEST produces expected results and not to MBP folder. I'm completely bewildered. What's inside MBP folder to cause the filter to break? It's a complete mystery!

I create filter_test_3.txt:

- /**/.DS_Store

+ /ssh/config
+ /ssh/vault/keys/**

- *

I put filter_test_3.txt in practice with my real life backup folder:

$ rclone lsf -R --filter-from /usr/local/etc/rclone/filter_test_3.txt Safe:/MBP
ssh/
ssh/config
ssh/vault/
ssh/vault/keys/
ssh/vault/keys/domus_rsa.pub

And it works! So, filter_test_2.txt does not work with my MBP folder.

Do you think such a file could conflict with filter_test_2.txt?

$ rclone ls Safe:MBP/Thinker --include '/.*'
        0 .Icon

I try to delete it, but can't:

$ rclone delete Safe:MBP/Thinker/.Icon
2019/06/06 10:58:12 ERROR : : error listing: directory not found
2019/06/06 10:58:12 ERROR : Attempt 1/3 failed with 2 errors and: directory not found
2019/06/06 10:58:12 ERROR : : error listing: directory not found
2019/06/06 10:58:12 ERROR : Attempt 2/3 failed with 2 errors and: directory not found
2019/06/06 10:58:12 ERROR : : error listing: directory not found
2019/06/06 10:58:12 ERROR : Attempt 3/3 failed with 2 errors and: directory not found
2019/06/06 10:58:12 Failed to delete: directory not found

ncw · June 6, 2019, 12:04pm

I've managed to reproduce that.

 rclone lsf -R --include '/a/{b,c/**}' .  --dump filters
--- start filters ---
--- File filter rules ---
+ ^a/(b|c/.*)$
- ^.*$
--- Directory filter rules ---
+ ^.*$        <---- this is wrong!
+ ^a/(b|c/.*)$
- ^.*$
--- end filters ---

There is a directory include filter which shouldn't be there.

Actually reading the code I see this

	// Can't deal with / or ** in {}
	tooHardRe = regexp.MustCompile(`{[^{}]*(\*\*|/)[^{}]*}`)

	if tooHardRe.MatchString(glob) {
		// Can't figure this one out so return any directory might match
		out = append(out, "/**")
		return out
	}

So when I wrote the code I punted this difficult case

The consequences are that you'll see every directory in the search path but you shouldn't see all the files.

That should produce a warning I suppose, or I should fix it...

Can you please make a new issue on github about this and we can look into fixing it - thanks.

system · June 9, 2019, 12:04pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.