How to sync and delete excluded directory

sp31415 · November 9, 2024, 3:33pm

What is the problem you are having with rclone?

I would like to sync a local directory to a cloud storage.
I used a filter file in order to select files I want on cloud storage.
I tried to exclude everything else. I succeeded to exclude some files (which has been removed from remote). I failed to exclude directory : they are kept as empty directory. I would like to get rid of these files.

Run the command 'rclone version' and share the full output of the command.

rclone v1.68.1

os/version: debian 12.7 (64 bit)
os/kernel: 6.1.0-26-amd64 (x86_64)
os/type: linux
os/arch: amd64
go/version: go1.23.1
go/linking: static
go/tags: none

Which cloud storage system are you using? (eg Google Drive)

union webdav mega

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone \
    -vv --config /dev/null \
    --mega-user "hidden" --mega-pass "hidden" \
    --webdav-vendor "owncloud" --webdav-url "https://drive.shadow.tech/remote.php/webdav" --webdav-user "hidden" --webdav-pass "hidden" \
    --union-upstreams '":webdav:" ":mega:"' \
    --union-create-policy "mfs" --union-cache-time 300 --union-min-free-space 100Mi \
    --bwlimit 61440k --dscp LE --transfers 1 --checkers 4 --stats 20m \
    --max-transfer 1300M --cutoff-mode=soft \
    --order-by size,asc --track-renames --track-renames-strategy leaf --size-only \
    --max-delete 50 --delete-after --delete-excluded \
    --filter-from /mnt/ER4MedTek/test-rclone/filters.txt \
    sync /mnt/ER4MedTek/test-rclone/ :union:test-rclone --dump filters

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

config are on CLI

empty

A log from the command that you were trying to run with the `-vv` flag

find /mnt/ER4MedTek/test-rclone/ | sort
/mnt/ER4MedTek/test-rclone/
/mnt/ER4MedTek/test-rclone/dira
/mnt/ER4MedTek/test-rclone/dira/dir1
/mnt/ER4MedTek/test-rclone/dira/dir1/zzzz_tree_20190406
/mnt/ER4MedTek/test-rclone/dira/dir1/zzzz_tree_20190823
/mnt/ER4MedTek/test-rclone/dirb
/mnt/ER4MedTek/test-rclone/dirb/dir2
/mnt/ER4MedTek/test-rclone/dirb/dir2/zzzz_tree_20180617
/mnt/ER4MedTek/test-rclone/dirb/dir2/zzzz_tree_20180828
/mnt/ER4MedTek/test-rclone/dirc
/mnt/ER4MedTek/test-rclone/dirc/dir3
/mnt/ER4MedTek/test-rclone/dirc/dir3/zzzz_tree_20180522
/mnt/ER4MedTek/test-rclone/dirc/dir3/zzzz_tree_20180601
/mnt/ER4MedTek/test-rclone/dirc/dir3/zzzz_tree_20180916
/mnt/ER4MedTek/test-rclone/dird
/mnt/ER4MedTek/test-rclone/dird/dir4
/mnt/ER4MedTek/test-rclone/dird/dir4/zzzz_tree_20180202
/mnt/ER4MedTek/test-rclone/dird/dir4/zzzz_tree_20180211
/mnt/ER4MedTek/test-rclone/filters.txt
/mnt/ER4MedTek/test-rclone/zzzz_tree_20180417
/mnt/ER4MedTek/test-rclone/zzzz_tree_20181213

cat /mnt/ER4MedTek/test-rclone/filters.txt
+ **/dir2/**
# exclude everything else
- **

2024/11/09 16:21:00 NOTICE: Automatically setting -vv as --dump is enabled
--- start filters ---
--- File filter rules ---
+ (^|/).*/dir2/.*$
- (^|/).*$
--- Directory filter rules ---
+ (^|/).*/dir2/.*/$
+ (^|/).*/dir2/$
+ (^|/).*/$
+ (^|/).*/dir2/.*$
- (^|/).*$
--- end filters ---
2024/11/09 16:21:00 INFO  : Starting bandwidth limiter at 60Mi Byte/s
2024/11/09 16:21:00 DEBUG : rclone: Version "v1.68.1" starting with parameters ["rclone" "-vv" "--config" "/dev/null" "--mega-user" "hidden" "--mega-pass" "hidden" "--webdav-vendor" "owncloud" "--webdav-url" "https://drive.shadow.tech/remote.php/webdav" "--webdav-user" "hidden" "--webdav-pass" "hidden" "--union-upstreams" "\":webdav:\" \":mega:\"" "--union-create-policy" "mfs" "--union-cache-time" "300" "--union-min-free-space" "100Mi" "--bwlimit" "61440k" "--dscp" "LE" "--transfers" "1" "--checkers" "4" "--stats" "20m" "--max-transfer" "1300M" "--cutoff-mode=soft" "--order-by" "size,asc" "--track-renames" "--track-renames-strategy" "leaf" "--size-only" "--max-delete" "50" "--delete-after" "--delete-excluded" "--filter-from" "/mnt/ER4MedTek/test-rclone/filters.txt" "sync" "/mnt/ER4MedTek/test-rclone/" ":union:test-rclone" "--dump" "filters"]
2024/11/09 16:21:00 DEBUG : Creating backend with remote "/mnt/ER4MedTek/test-rclone/"
2024/11/09 16:21:00 DEBUG : Using config file from ""
2024/11/09 16:21:00 DEBUG : fs cache: renaming cache item "/mnt/ER4MedTek/test-rclone/" to be canonical "/mnt/ER4MedTek/test-rclone"
2024/11/09 16:21:00 DEBUG : Creating backend with remote ":union:test-rclone"
2024/11/09 16:21:00 DEBUG : :union: detected overridden config - adding "{8WRJv}" suffix to name
2024/11/09 16:21:00 DEBUG : Creating backend with remote ":mega:"
2024/11/09 16:21:00 DEBUG : :mega: detected overridden config - adding "{N4EtT}" suffix to name
2024/11/09 16:21:00 DEBUG : Creating backend with remote ":webdav:"
2024/11/09 16:21:00 DEBUG : :webdav: detected overridden config - adding "{Ed1v0}" suffix to name
2024/11/09 16:21:00 DEBUG : found headers: 
2024/11/09 16:21:00 DEBUG : fs cache: renaming cache item ":webdav:" to be canonical ":webdav{Ed1v0}:"
2024/11/09 16:21:00 DEBUG : Creating backend with remote ":webdav:test-rclone"
2024/11/09 16:21:00 DEBUG : :webdav: detected overridden config - adding "{Ed1v0}" suffix to name
2024/11/09 16:21:00 DEBUG : found headers: 
2024/11/09 16:21:00 DEBUG : fs cache: renaming cache item ":webdav:test-rclone" to be canonical ":webdav{Ed1v0}:test-rclone"
2024/11/09 16:21:00 DEBUG : fs cache: renaming cache item ":mega:" to be canonical ":mega{N4EtT}:"
2024/11/09 16:21:00 DEBUG : Creating backend with remote ":mega:test-rclone"
2024/11/09 16:21:00 DEBUG : :mega: detected overridden config - adding "{N4EtT}" suffix to name
2024/11/09 16:21:00 DEBUG : fs cache: renaming cache item ":mega:test-rclone" to be canonical ":mega{N4EtT}:test-rclone"
2024/11/09 16:21:00 DEBUG : union root 'test-rclone': actionPolicy = *policy.EpAll, createPolicy = *policy.Mfs, searchPolicy = *policy.FF
2024/11/09 16:21:00 DEBUG : fs cache: renaming cache item ":union:test-rclone" to be canonical ":union{8WRJv}:test-rclone"
2024/11/09 16:21:01 DEBUG : dirb/dir2/zzzz_tree_20180828: Sizes identical
2024/11/09 16:21:01 DEBUG : dirb/dir2/zzzz_tree_20180828: Unchanged skipping
2024/11/09 16:21:01 DEBUG : dirb/dir2/zzzz_tree_20180617: Sizes identical
2024/11/09 16:21:01 DEBUG : dirb/dir2/zzzz_tree_20180617: Unchanged skipping
2024/11/09 16:21:01 INFO  : union root 'test-rclone': Making map for --track-renames
2024/11/09 16:21:01 INFO  : union root 'test-rclone': Finished making map for --track-renames
2024/11/09 16:21:01 DEBUG : union root 'test-rclone': Waiting for checks to finish
2024/11/09 16:21:01 DEBUG : union root 'test-rclone': Waiting for renames to finish
2024/11/09 16:21:01 DEBUG : union root 'test-rclone': Waiting for transfers to finish
2024/11/09 16:21:01 DEBUG : Waiting for deletions to finish
2024/11/09 16:21:01 INFO  : There was nothing to transfer
2024/11/09 16:21:01 INFO  : 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Checks:                 2 / 2, 100%
Elapsed time:         0.7s

2024/11/09 16:21:01 DEBUG : 10 go routines active

lsf :union:test-rclone
dira/
dirb/
dirc/
dird/
2024/11/09 16:18:03 INFO  :

I would like to delete dira dirc dird which I expect to match "- **"

kapitainsky · November 9, 2024, 4:20pm

--delete-excluded only applies to files.

Second step is needed here to delete things from source. maybe rclone delete with the same filter followed by rclone rmdirs Or maybe only rclone rmdirs as files should be already deleted.

system · December 9, 2024, 4:20pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.