Includes empty source folders when using slashes in glob choice patterns

Suspected Bug
1

Hi, thank you so much for rclone, it solves a lot of my problems!

With making extensive use of filter patterns I came across extra empty folders in the output.

I can work around it at the moment, but I was wondering if I misunderstood some part of glob pattern matching (possibly causing more surprises down the line) or if it is just a minor bug.

I tried to give a simple example for easy check. Actually I have many filters in a (partially generated) filter file among other patterns and I am not syncing locally.

What is the problem you are having with rclone?

Given two files

./source/included/included.txt
./source/other/child/other.log

Running rclone sync (see command below) to an empty ./target folder with --create-empty-src-dirs flag creates directories for excluded (but existing) paths as well. This does not seem to reproduce when the glob options do not contain slashes.

Expected result:

./target/included/included.txt

Actual result:

./target/other/child/
./target/included/included.txt

What is your rclone version (output from rclone version)

rclone v1.53.2-DEV

Also reproduced on rclone v1.53.1-DEV and rclone-v1.54.0-beta.4868.0e7fc7613-osx-amd64.

Which OS you are using and how many bits

Mac OS 64 bit 10.15.7

Which cloud storage system are you using?

It is reproducible locally as well.

The command you were trying to run

rclone sync ./source ./target --filter='+ /{included/included.txt,nonexistent/nonexistent.txt}' --filter='- **' --create-empty-src-dirs

It is reproducible with --dry-run as well.

The rclone config contents with secrets removed.

It is reproducible without config.

A log from the command with the -vv flag

2020/10/29 23:42:14 DEBUG : Creating backend with remote "./source"
2020/10/29 23:42:14 NOTICE: Config file "/Users/madisp/.config/rclone/rclone.conf" not found - using defaults
2020/10/29 23:42:14 DEBUG : fs cache: renaming cache item "./source" to be canonical "/Users/madisp/Desktop/source"
2020/10/29 23:42:14 DEBUG : Creating backend with remote "./target"
2020/10/29 23:42:14 DEBUG : fs cache: renaming cache item "./target" to be canonical "/Users/madisp/Desktop/target"
2020/10/29 23:42:14 DEBUG : other/child/other.log: Excluded
2020/10/29 23:42:14 DEBUG : Local file system at /Users/madisp/Desktop/target: Waiting for checks to finish
2020/10/29 23:42:14 DEBUG : Local file system at /Users/madisp/Desktop/target: Waiting for transfers to finish
2020/10/29 23:42:14 DEBUG : included/included.txt: MD5 = d41d8cd98f00b204e9800998ecf8427e OK
2020/10/29 23:42:14 INFO  : included/included.txt: Copied (new)
2020/10/29 23:42:14 DEBUG : other/child: Making directory
2020/10/29 23:42:14 DEBUG : Local file system at /Users/madisp/Desktop/target: copied 1 directories
2020/10/29 23:42:14 DEBUG : Waiting for deletions to finish
2020/10/29 23:42:14 INFO  :
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         0.0s

2020/10/29 23:42:14 DEBUG : 5 go routines active

As said before, it is a minor issue but I'm not completely sure if I'm misunderstanding the purpose of creating empty dirs or not.

Thanks for your help

2

Rclone filters describe what you want to happen to files. The directories less important in filter world.

Rclone will skip a directory if the filters don't include it, eg --exclude "/dir/".

However if rclone has to recurse into a directory to see if there are files it needs filter then rclone will create that directory even if there are no files in it.

Concrete example

.
├── a.txt
├── b.txt
├── dir
│   ├── c.txt
│   └── d.txt
└── e.jpg

Notice how dir/ is included

$ rclone lsf . --include "*txt"
rclone lsf -R . --include "*.txt"
a.txt
b.txt
dir/
dir/c.txt
dir/d.txt

And also here

$  rclone lsf -R . --include "*.jpg"
dir/
e.jpg

But not here

 rclone lsf -R . --exclude "dir/"
a.txt
b.txt
e.jpg

I hope that helps!

Note that rclone has an rclone rmdirs command for removing (recursively) empty directories.

1 Like
3

Thanks for the help!
The behavior makes sense now and I can work around it consistently.
Also good idea to use lsf for testing the patterns!

1 Like
4

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.