Excluded files are excluded also in the destination directory and never removed

What is the problem you are having with rclone?

When using rclone sync with --exclude flag, the excluded files are excluded not only in the source directory, but also in the destination directory.
This is my layout:

/tmp/rclonetest$ tree
.
├── dst
│   └── tmp
│       └── file.log
└── src
    ├── dir2
    │   └── file2.txt
    └── file1.txt

After running rclone sync -v src dst --exclude=*.log the layout is:

/tmp/rclonetest$ tree
.
├── dst
│   ├── dir2
│   │   └── file2.txt
│   ├── file1.txt
│   └── tmp
│       └── file.log
└── src
    ├── dir2
    │   └── file2.txt
    └── file1.txt

Logs say:

...
2024/02/06 22:24:20 INFO  : tmp: Removing directory
...

but the directory tmp is not removed at all. If I launch the same command again, I get the same output, but the tmp directory is never removed.

Run the command 'rclone version' and share the full output of the command.

rclone v1.65.2

  • os/version: ubuntu 22.04 (64 bit)
  • os/kernel: 6.5.0-15-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.21.6
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Local.

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync -v src dst --exclude=*.log

The rclone config contents with secrets removed.

empty

A log from the command with the -vv flag

2024/02/06 22:24:20 INFO  : file1.txt: Copied (new)
2024/02/06 22:24:20 INFO  : dir2/file2.txt: Copied (new)
2024/02/06 22:24:20 INFO  : tmp: Removing directory
2024/02/06 22:24:20 INFO  : 
Transferred:   	         12 B / 12 B, 100%, 0 B/s, ETA -
Deleted:                0 (files), 1 (dirs)
Transferred:            2 / 2, 100%
Elapsed time:         0.0s

This part is expected. The same filters have to apply on both sides, because otherwise if a file was excluded on the source but present on the dest, rclone would think it is supposed to be deleted during the sync.

I'm pretty sure that what's happening here is that rclone is trying to rmdir the directory, but this is silently failing because the directory isn't actually empty. It appears empty because file.log is excluded with filters, but the local backend knows that file.log is actually there, and therefore refuses to remove the directory. (If you add -vv I think you should see a debug log confirming this.)

This is an issue with no perfect solution at the moment, although I actually submitted a PR recently that proposes a fix, so hopefully there will be a better solution soon:

In the meantime, what I'd recommend is either to delete the file (remove filter or manually delete), move it out of your tree, or keep it as is but accept that you won't be able to delete that directory. (I run into this myself constantly with .DS_Store files on macOS -- and I've generally found that trying to exclude them with filters causes more problems than it solves. That should change once the PR is merged.)

1 Like

If you're feeling adventurous, there's also the --delete-excluded flag... but it can really be quite destructive so I would urge you to make sure you understand how it works before trying it. And --dry-run first! :sweat_smile: (In most cases, I think you are better off just manually deleting the problematic file.)

I will experiment with this option, I had overlooked it. In the mean time I will be waiting for your PR to be merged. Thank you

1 Like

Actually, running the command with the -vv flag the output is:

...
2024/02/07 22:24:41 INFO  : tmp: Removing directory
2024/02/07 22:24:41 DEBUG : tmp: Failed to Rmdir: remove /tmp/rclonetest/dst/tmp: directory not empty
2024/02/07 22:24:41 DEBUG : Local file system at /tmp/rclonetest/dst: failed to delete 1 directories
...

This is what I predicted.

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.