Extension or misunderstanding --exclude-if-present

Help and Support
1

For a tape-based backup job that can have changes, I only want to back up "new" items but in this instance date-based filtering cannot be used.

I had planned to use rclone ls --exclude-if-present feature but it seems either I misunderstand it or it is not the best tool.

Essentially there are directories and from a given point in the hierarchy I'd run my rclone backup command to spool to tape. BUT I would NOT want a given directory [and its subs thereof] copied of a lock file is there.

So /root/dog/files... would be accepted but /root/cat/files would NOT backup /root/cat/* IF there was a lock file in /root/cat, e.g. .ignore

But in all my tests so far it seems to want to block everything from root, since ONE of the subdirs contains the lock file.

So if you see

~/storage/DB1-ENC/test  tree -a
.
├── 1
│   ├── 1.txt
│   └── 2
│       ├── 2.txt
│       └── 3
│           ├── 3.txt
│           └── ignore.ignore
└── temp
    ├── 1
    ├── 2
    │   └── deep.txt
    └── .ignore

That temp dir should copy /deep/2 but it won't as this is the output I get as a test:

~/storage/DB1-ENC/test rclone ls DB1-ENC:test --exclude-if-present .ignore 0 1/1.txt 0 1/2/2.txt 0 1/2/3/3.txt 0 1/2/3/ignore.ignore

Would welcome guidance!

Run the command 'rclone version' and share the full output of the command.

rclone v1.62.2

  • os/version: debian 11.7 (64 bit)
  • os/kernel: 5.10.0-25-amd64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.20.2
  • go/linking: static
  • go/tags: none

I tested it on my laptop with 1.64.1 too, unless that was used error.

Which cloud storage system are you using? (eg Google Drive)

Dropbox

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Shown in above example.

The rclone config contents with secrets removed.

I am presuming it isnot needed in this instance, everything else works

A log from the command with the -vv flag

6:44 ~/storage/DB1-ENC/test  rclone ls DB1-ENC:test --exclude-if-present .ignore -vv
2023/09/17 06:44:35 DEBUG : rclone: Version "v1.62.2" starting with parameters ["rclone" "ls" "DB1-ENC:test" "--exclude-if-present" ".ignore" "-vv"]
2023/09/17 06:44:35 DEBUG : Creating backend with remote "DB1-ENC:test"
2023/09/17 06:44:35 DEBUG : Using config file from "/home/xxxxx/.config/rclone/rclone.conf"
2023/09/17 06:44:35 DEBUG : Creating backend with remote "DB1:rc/enc/0bcokthicd02sn38qhrelbf03o"
2023/09/17 06:44:36 DEBUG : temp: Excluded
        0 1/1.txt
        0 1/2/2.txt
        0 1/2/3/3.txt
        0 1/2/3/ignore.ignore
2023/09/17 06:44:38 DEBUG : 6 go routines active
2023/09/17 06:44:38 INFO  : Dropbox root 'rc/enc/0bcokthicd02sn38qhrelbf03o': Committing uploads - please wait...
6:44 ~/storage/DB1-ENC/test  ```
2

If you put .ignore file in /temp directory you affectively create --exclude /temp/** rule which excludes directory and all its content (including subdirectories)

Not sure where you see this behaviour. Your example shows exactly something opposite.
You have .ignore file in /temp folder and it does not block content of /1 folder.

What exactly you want to ignore - as from your description you want to backup all files present.

3

Thank you the response and my apologies for my delayed answer (illness).

I have re-made the test in case my original was affected by my pending illness and blocked head.

So everything [for this example] is in one directory, split into two sub-directories, named easily!

@M01 testdir % tree -a
.

|-- test-exclude
|   `-- ignore-me.ignore
`-- test-no-exclude
    `-- no-exclude.txt

The objective is that the file .ignore would NOT be processed. Even better if there was an option to then say ignore the directory name too [test-exclude] as that would be nice for a second purpose. But one step at a time.

rclone ls . --exclude-if-present .ignore
       52 test-no-exclude/no-exclude.txt
       51 test-exclude/ignore-me.ignore

The same happens if I double quote .ignore. It lists the .ignore file.

It even does it with copy [tested in case for some reason ls was treating it differently)

rclone copy . REMOTE:Test --dry-run --exclude-if-present .ignore
2023/09/23 09:45:24 NOTICE: test-no-exclude/no-exclude.txt: Skipped copy as --dry-run is set (size 52)
2023/09/23 09:45:24 NOTICE: test-exclude/ignore-me.ignore: Skipped copy as --dry-run is set (size 51)
2023/09/23 09:45:24 NOTICE:
Transferred: 103 B / 103 B, 100%, 0 B/s, ETA -
Transferred: 2 / 2, 100%
Elapsed time: 0.9s

So... I am stuck. You mention otherwise patterns but that side of thing I've never got my head around it and, if that fixes the problem, I'd be grateful for some practical show 'n' tell!

The objectives:

Objective 1: I have items to back up to long-term storage. I intend to put an .ignore [or .lock] small text file in the root of the directory to say DO NOT BACK UP THIS SUB-DIRECTORY OR DEEPER.

e..g /Files/Pictures/2001/Holiday/there/are/many/subdirs

I would put say .lock in /File/Pictures/2001/ignore.lock and then it would know not to back up that dir [ideally inc. header dir of 2001 AND NOT PROCESS ANYTHING recursive to this.

There would be hundreds or more of these directories, so tracking each and every one in a file would be a major job in itself and I fear would add overload to the process [or rclone] but if needs must..

Does this explain what I want to do, how I've tried, how it fails for me and give light to any solution you can kindly think of?

TIA!

4

Have a look at this example:

$ tree -a .
.
├── A
│   ├── .ignore
│   ├── A1
│   │   ├── A11
│   │   │   └── testA11.file
│   │   └── testA1.file
│   ├── A2
│   │   └── testA2.file
│   └── testA.file
└── B
    ├── B1
    │   └── testB1.file
    ├── B2
    │   └── testB2.file
    └── testB.file

8 directories, 8 files

There is .ignore file in ./A directory so all ./A and below is excluded

$ rclone ls . --exclude-if-present .ignore
        0 B/testB.file
        0 B/B2/testB2.file
        0 B/B1/testB1.file

Now if I put .ignore in ./A1 instead:

$ tree -a .
.
├── A
│   ├── A1
│   │   ├── .ignore
│   │   ├── A11
│   │   │   └── testA11.file
│   │   └── testA1.file
│   ├── A2
│   │   └── testA2.file
│   └── testA.file
└── B
    ├── B1
    │   └── testB1.file
    ├── B2
    │   └── testB2.file
    └── testB.file

8 directories, 8 files

./A1 and below is excluded:

$ rclone ls . --exclude-if-present .ignore
        0 B/testB.file
        0 A/testA.file
        0 B/B2/testB2.file
        0 A/A2/testA2.file
        0 B/B1/testB1.file

Simply when I use --exclude-if-present .ignore any directory (and below) containing .ignore file is excluded. This is exactly what matches your Objective1

Not sure what is not clear...

Maybe problem is here?:

ignore-me.ignore file is not the same as .ignore file you use in your rule --exclude-if-present .ignore

So when you run:

Nothing is excluded. As there is no .ignore file anywhere

If you want to use ignore-me.ignore file then use --exclude-if-present ignore-me.ignore. You can use any file name here, e.g.: --exclude-if-present TestFile. This file name is up to you.

5

Thank you. I never got a mail alert for this so it was by chance I came back to check.

For some reason, that I can't explain, I had got it into my head that the exclude pattern would be somehow a wildcard [so *.ignore] rather than an absolute.

I can't necessarily blame the documentation for reader error, but maybe a slight emphasis could help!

So now in my "real-world" test if I '''rclone ls . --exclude-if-present Archived.lock -vv''' against a big directory of directories which have INSIDE them a file called Archived.lock I get '''2023/09/26 07:37:49 DEBUG : Att angöra en brygga (1965) [SE]: Excluded''' [I left the verbose flag in to see the processing and be certain! This is exactly the behaviour I want, so it would [in this case] DISPLAY any directories that DID NOT have Archived.lock.

I am very grateful for you pushing me in the right direction. Sometimes with my sight in particular I might miss the damn obvious.

BW

1 Like
6

Documentation PRs are always welcomed. I agree it can always be better.