--fast-list changes behavior of --exclude

Suspected Bug
1

What is the problem you are having with rclone?

rclone sync deleted files under a excluded directory when using fast-list, even without --delete-excluded

What is your rclone version (output from rclone version)

1.48.0 and master/def7909

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Ubuntu 16.04.6 64bit

Which cloud storage system are you using? (eg Google Drive)

Google Drive

I did six experiments, with/without --fast-list and --exclude f, d/, d/**.

Setup

  1. Generate two files: tmp/f and tmp/d/df.
  2. Sync: rclone sync tmp gdrive:tmp

Results

Excluding file is OK, however, the output is different.

$ rclone sync -nvv tmp gdrive:tmp --exclude f
DEBUG : f: Excluded
DEBUG : f: Excluded
$ rclone sync -nvv tmp gdrive:tmp --exclude f --fast-list
DEBUG : f: Excluded
DEBUG : f: Excluded from sync (and deletion)

When --exclude d/ --fast-list, the messages are inconsistent: rclone is not deleting and deleting the directory.

$ rclone sync -nvv tmp gdrive:tmp --exclude d/
DEBUG : d: Excluded
DEBUG : d: Excluded
$ rclone sync -nvv tmp gdrive:tmp --exclude d/ --fast-list
DEBUG : d: Excluded
DEBUG : d: Excluded from sync (and deletion)  # <- not deleting `d/`
NOTICE : d/df: Not deleting as --dry-run
NOTICE : d: Not deleting as dry run is set    # <- deleting `d/`

When --fast-list --exclude 'd/**', they're consistent again:

$ rclone sync -nvv tmp gdrive:tmp --exclude d/
DEBUG : d: Excluded
DEBUG : d: Excluded
$ rclone sync -nvv tmp gdrive:tmp --exclude d/ --fast-list
DEBUG : d: Excluded
DEBUG : d: Excluded from sync (and deletion)
DEBUG : d/df: Excluded from sync (and deletion)

Question: How should I exclude a directory? Always use d/** ?
And how to exclude directory content but include directory itself? Use --filter?

A random guess:
I didn't dig into the code, but I guess this is cause by the exclude process difference between file tree structure (removing TreeNode(d/) implicitly removes TreeNode(d/df) ) and file list structure (ListNode(d/) and ListNode(d/df) are independent).

2

That is the correct answer, but it isn't satisfactory IMHO

directory/ just refers to the directory, however when filtering it is used to prune the directory without --fast-list and it isn't with --fast-list - I think that is where the inconsistency is coming from.

So I think this is a bug, but one which would be fixed by making directory/ == directory/** which I've been thinking of doing anyway since it causes user confusion.

You will certainly need to use --filter.

Perhaps something like

- dir/**
+ dir/

Thoughts?

3

I think that making dir/ becomes dir/** in --exclude is OK, as --filter is already there for fine-tuning filtering.

4

Do you think you could turn this into a new issue on github then we can do the dir/** to dir/ change which I think will fix your issue.

Thanks

5

OK. Here is it: https://github.com/ncw/rclone/issues/3375

Thanks

1 Like
6

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.