Is it possible to exclude files/folders from being deleted on the destination?

What is the problem you are having with rclone?

When syncing from a source to a destination I would like to exclude certain files from being deleted on the remote, in particular auto-generated *.sdr/ folders on my eReader. I've tried using --exclude "**/*.sdr/", however this has had no effect as it continues to (not actually as --dry-run) delete files on the destination.

The intention is that it DOES delete OTHER files that are not excluded. For example if I rename a .pdf file it should delete the original filename (while cloning the new name). But it shouldn't delete the excluded *.sdr/ files on the destination.

What is your rclone version (output from rclone version)

rclone v1.55.1
- os/type: linux
- os/arch: amd64
- go/version: go1.16.3
- go/linking: static
- go/tags: none

Which OS you are using and how many bits (eg Windows 7, 64 bit)

ChromeOS 91/Crostini Linux which is a modified version of Debian 10.

Which cloud storage system are you using? (eg Google Drive)

Onedrive is the source, however the destination is just a directory (mounted through ChromeOS).

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync --dry-run -P "onedrive:/Course Materials/UC/MATH416/" . --exclude "/**/*.sdr/"
# Also tried with "**/*.sdr", "**/*.sdr/", "/**/*.sdr" so I don't think it's to do with the pattern

The rclone config contents with secrets removed.

[onedrive]
type = onedrive
region = global
token = <<OMITTED>>
drive_id = <<OMITTED>>
drive_type = personal

[googleDrive]
type = drive
scope = drive
token = <<OMITTED>>

A log from the command with the -vv flag

<7>DEBUG : Using config file from "/home/jamesernator/.config/rclone/rclone.conf"
<7>DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "sync" "--dry-run" "-P" "onedrive:/Course Materials/UC/MATH416/" "." "--exclude" "/**/*.sdr/" "-vv"]
<7>DEBUG : rclone: systemd logging support activated
<7>DEBUG : Creating backend with remote "onedrive:/Course Materials/UC/MATH416/"
<7>DEBUG : fs cache: renaming cache item "onedrive:/Course Materials/UC/MATH416/" to be canonical "onedrive:Course Materials/UC/MATH416"
<7>DEBUG : Creating backend with remote "."
<7>DEBUG : fs cache: renaming cache item "." to be canonical "/mnt/chromeos/removable/KOBOeReader/Course Materials/MATH416"
2021-07-08 23:04:13 DEBUG : 416.pdf: Modification times differ by -1s: 2021-03-04 02:42:01 +0000 UTC, 2021-03-04 15:42:00 +1300 NZDT
2021-07-08 23:04:13 DEBUG : Exam.pdf: Size and modification time the same (differ by 0s, within tolerance 1s)
2021-07-08 23:04:13 DEBUG : Exam.pdf: Unchanged skipping
2021-07-08 23:04:13 DEBUG : Local file system at /mnt/chromeos/removable/KOBOeReader/Course Materials/MATH416: Waiting for checks to finish
2021-07-08 23:04:13 DEBUG : 416.pdf: SHA-1 = ad095e6da60f40fdb389b535cd900c50666eb0fb OK
2021-07-08 23:04:13 NOTICE: 416.pdf: Skipped update modification time as --dry-run is set (size 374.751k)
2021-07-08 23:04:13 DEBUG : 416.pdf: Unchanged skipping
2021-07-08 23:04:13 DEBUG : Local file system at /mnt/chromeos/removable/KOBOeReader/Course Materials/MATH416: Waiting for transfers to finish
2021-07-08 23:04:13 DEBUG : Waiting for deletions to finish
2021-07-08 23:04:13 NOTICE: 416.sdr/metadata.pdf.lua: Skipped delete as --dry-run is set (size 3.157k)
2021-07-08 23:04:13 NOTICE: 416.sdr: Skipped remove directory as --dry-run is set
2021-07-08 23:04:13 DEBUG : Local file system at /mnt/chromeos/removable/KOBOeReader/Course Materials/MATH416: deleted 1 directories
2021-07-08 23:04:13 INFO  : There was nothing to transfer
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:                 3 / 3, 100%
Deleted:                1 (files), 1 (dirs)
Elapsed time:         3.2s
<5>NOTICE: 
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:                 3 / 3, 100%
Deleted:                1 (files), 1 (dirs)
Elapsed time:         3.2s

<7>DEBUG : 4 go routines active

It's easiest to test with rclone ls.

Can you share the rclone ls output of the remote as I'm trying to follow what you want to delete and not delete and it's tough from the sync output to figure that out.

You want to exclude any directory and it's contents with "sdr" in the name of the directory?

So:

/test/example - stays
/123.sdr/deletedfile - gets deleted?

The directory shown is just a small sample subset of what I want to sync, but it gives a simple example of what goes wrong. Basically the source directory looks like this:

onedrive:/Course Materials/UC/MATH416/
  - 416.pdf
  - Exam.pdf

The destination directory looks like this:

/mnt/chromeos/removable/KOBOeReader/Course Materials/MATH416/
  - 416.pdf
  # This directory is autogenerated on the eReader and contains data such as current read position
  # settings for the file, etc etc
  - 416.sdr/ 
    - metadata.pdf.lua
  - Exam.pdf

What I want to happen is that rclone sync keeps up to date all files in the directories (which is a massive superset of that one). For example suppose I renamed Exam.pdf to Examination.pdf then the source directory would be:

onedrive:/Course Materials/UC/MATH416/
  - 416.pdf
  - Examination.pdf

Now when syncing to the destination I would like the destination to become:

/mnt/chromeos/removable/KOBOeReader/Course Materials/MATH416/
  - 416.pdf
  # This directory is autogenerated on the eReader and contains data such as current read position
  # settings for the file, etc etc
  # This doesn't exist in the source, but it should NOT be deleted
  - 416.sdr/ 
    - metadata.pdf.lua
  # Notice this file has been renamed, so it would be synced and Exam.pdf
  # has been deleted as per usual sync
  - Examination.pdf

Note that rclone copy does not work for this, as with copy it would not delete the Exam.pdf, instead there would be two duplicate files on the destination, Exam.pdf and Examination.pdf.

The problem is essentially that I would like to ignore files on the destination, but passing --exclude "**/*.sdr/" (or any variants mentioned above) does not change anything. My presumption is that exclude only applies to files in the source not the destination.

As such this question (possibly a suggestion if it's impossible) is as to whether or not there is a way to exclude files on the destination from being deleted during syncing, note that files not in the excluded should be deleted as per usual (e.g. the example above of renaming Exam.pdf -> Examination.pdf should delete Exam.pdf on the destination as usual).

You exclude files and not directories. So if you want to exclude files in a directory you need to match the file and path. And yes you'd want to use sync.

**/*.sdr/**

Unless I'm reading what you're asking wrong. Warning I haven't finished my coffee. :crazy_face:

ref: Rclone Filtering

Edit: The first slash though will cause you issues if its in the base. You may want this.

--exclude="**.sdr/**"

dell-xps:~$ rclone ls dest/MATH416 --exclude="**/*.sdr/**"
        2 416.pdf
        2 Exam.pdf
        2 416.sdr/metadata.pdf.lua
dell-xps:~$ rclone ls dest/MATH416 --exclude="**.sdr/**"
        2 416.pdf
        2 Exam.pdf
dell-xps:~$ rclone sync --dry-run MATH416 dest/MATH416 --exclude="**.sdr/**"
2021/07/08 08:40:05 NOTICE: 416.pdf: Skipped update modification time as --dry-run is set (size 2)
2021/07/08 08:40:05 NOTICE: Exam.pdf: Skipped update modification time as --dry-run is set (size 2)
2021/07/08 08:40:05 NOTICE: 
Transferred:   	          0 / 0 Byte, -, 0 Byte/s, ETA -
Checks:                 2 / 2, 100%
Elapsed time:         0.0s

dell-xps:~$ rclone sync  MATH416 dest/MATH416 --exclude="**.sdr/**"
dell-xps:~$ find MATH416
MATH416
MATH416/Exam.pdf
MATH416/416.pdf
dell-xps:~$ find dest
dest
dest/MATH416
dest/MATH416/Exam.pdf
dest/MATH416/416.sdr
dest/MATH416/416.sdr/metadata.pdf.lua
dest/MATH416/416.pdf
dell-xps:~$ mv MATH416/Exam.pdf MATH416/Examination.pdf
dell-xps:~$ rclone sync  MATH416 dest/MATH416 --exclude="**.sdr/**"
dell-xps:~$ find dest
dest
dest/MATH416
dest/MATH416/416.sdr
dest/MATH416/416.sdr/metadata.pdf.lua
dest/MATH416/416.pdf
dest/MATH416/Examination.pdf

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.