Deleting of existent excluded files

What is the problem you are having with rclone?

Existent excluded files aren't deleted with the "--delete-excluded" flag. I'm not sure if it's a bug or if I'm doing something wrong.

Run the command 'rclone version' and share the full output of the command.

rclone v1.65.2

  • os/version: linuxmint 20.3 (64 bit)
  • os/kernel: 5.4.0-171-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.21.6
  • go/linking: static
  • go/tags: none

AND

rclone v1.65.2

  • os/version: Microsoft Windows 10 Pro 22H2 (64 bit)
  • os/kernel: 10.0.19045.3803 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.21.6
  • go/linking: static
  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Dropbox

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone bisync "dropbox:/New MyProject Team/test/season_1/job_1" "/home/user/temp/MyProject/dropbox_editors/job_1_[42]" --create-empty-src-dirs --ignore-listing-checksum --exclude "*.jpg" --delete-excluded --log-level DEBUG

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[dropbox]
type = dropbox
client_id = XXX
client_secret = XXX
token = XXX

A log from the command that you were trying to run with the -vv flag

<7>DEBUG : rclone: Version "v1.65.2" starting with parameters ["/home/user/opt/rclone" "bisync" "dropbox:/New MyProject Team/test/season_1/job_1" "/home/user/temp/MyProject/dropbox_editors/job_1_[42]" "--create-empty-src-dirs" "--ignore-listing-checksum" "--exclude" "*.jpg" "--delete-excluded" "--log-level" "DEBUG"]
<7>DEBUG : rclone: systemd logging support activated
<7>DEBUG : Creating backend with remote "dropbox:/New MyProject Team/test/season_1/job_1"
<7>DEBUG : Using config file from "/home/user/.config/rclone/rclone.conf"
<7>DEBUG : Dropbox root '': Using root namespace "2812304097"
<7>DEBUG : fs cache: renaming cache item "dropbox:/New MyProject Team/test/season_1/job_1" to be canonical "dropbox:New MyProject Team/test/season_1/job_1"
<7>DEBUG : Creating backend with remote "/home/user/temp/MyProject/dropbox_editors/job_1_[42]"
<5>NOTICE: bisync is EXPERIMENTAL. Don't use in production!
<7>DEBUG : Lock file created: /home/user/.cache/rclone/bisync/dropbox_New_MyProject_Team_test_season_1_job_1..home_user_temp_MyProject_dropbox_editors_job_1_[42].lck
<6>INFO  : Synching Path1 "dropbox:New MyProject Team/test/season_1/job_1/" with Path2 "/home/user/temp/MyProject/dropbox_editors/job_1_[42]/"
<6>INFO  : Path1 checking for diffs
<7>DEBUG : t20e/wallhaven-1jxlz3.jpg: Excluded
<7>DEBUG : t20e/wallhaven-4y56vd.jpg: Excluded
<6>INFO  : - Path1    File was deleted                    - t20e/wallhaven-1jxlz3.jpg
<6>INFO  : - Path1    File was deleted                    - t20e/wallhaven-4y56vd.jpg
<6>INFO  : Path1:    2 changes:    0 new,    0 newer,    0 older,    2 deleted
<6>INFO  : Path2 checking for diffs
<6>INFO  : - Path2    File was deleted                    - t20e/wallhaven-1jxlz3.jpg
<6>INFO  : - Path2    File was deleted                    - t20e/wallhaven-4y56vd.jpg
<6>INFO  : Path2:    2 changes:    0 new,    0 newer,    0 older,    2 deleted
<6>INFO  : Applying changes
<7>DEBUG : loading listing for path 1 at: /home/user/.cache/rclone/bisync/dropbox_New_MyProject_Team_test_season_1_job_1..home_user_temp_MyProject_dropbox_editors_job_1_[42].path1.lst-new
<7>DEBUG : found a dir: t20e
<7>DEBUG : not a dir: t20e/wallhaven-2ejw2g.png
<7>DEBUG : found a dir: t20e/tr91
<7>DEBUG : not a dir: t20e/tr91/ewrwaerawer.txt
<7>DEBUG : found a dir: t20e/tr91/k%20l o
<7>DEBUG : not a dir: t20e/tr91/k%20l o/ewrwaerawer
<7>DEBUG : loading listing for path 2 at: /home/user/.cache/rclone/bisync/dropbox_New_MyProject_Team_test_season_1_job_1..home_user_temp_MyProject_dropbox_editors_job_1_[42].path2.lst-new
<7>DEBUG : found a dir: t20e
<7>DEBUG : not a dir: t20e/wallhaven-2ejw2g.png
<7>DEBUG : found a dir: t20e/tr91
<7>DEBUG : not a dir: t20e/tr91/ewrwaerawer.txt
<7>DEBUG : found a dir: t20e/tr91/k%20l o
<7>DEBUG : not a dir: t20e/tr91/k%20l o/ewrwaerawer
<6>INFO  : Updating listings
<6>INFO  : Validating listings for Path1 "dropbox:New MyProject Team/test/season_1/job_1/" vs Path2 "/home/user/temp/MyProject/dropbox_editors/job_1_[42]/"
<7>DEBUG : Lock file removed: /home/user/.cache/rclone/bisync/dropbox_New_MyProject_Team_test_season_1_job_1..home_user_temp_MyProject_dropbox_editors_job_1_[42].lck
<6>INFO  : Bisync successful
<6>INFO  : 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         6.2s

<7>DEBUG : 7 go routines active
<6>INFO  : Dropbox root 'New MyProject Team/test/season_1/job_1': Committing uploads - please wait...

I use bisync to synchronize this folder and everything works well.
I tried to exclude files for the first sync and all worked well. But if I try to use an exclude filter with the "--delete-excluded" flag while syncing an existing job it doesn't delete any file in destination.
So, here it says that the jpg files have been deleted, but they are still available on the local machine. Is it OK?
Thanks.

Short answer: I would strongly advise against using --delete-excluded in bisync, as it will probably not do what you are expecting. Longer explanation below.

In the version you are using (v1.65.2), what is happening is:

  • The exclusion filter causes Bisync to think the file has already been deleted on both sides, and therefore Bisync thinks there's nothing further it needs to do (no use trying to delete something that's already deleted)

  • If there are any other new/changed files during the same run, Bisync copies them with an internal call roughly equivalent to rclone copy src dst --files-from [files]. --delete-excluded is still ignored here because rclone copy never deletes things on the dest (as opposed to rclone sync, which does.)

In the upcoming v1.66, Bisync's internal calls use sync instead of copy, meaning that --delete-excluded will no longer be ignored when there are files to copy. However, I would still strongly advise against using it, as it will work, but probably not in the way that you are expecting!

The thing is -- given the bidirectional nature of bisync, both sides are sources and both sides are destinations, at some point. As I mentioned above, bisync internally uses --filter-from to limit the sync calls to only the files which bisync has already determined need to be copied or deleted. (Among other things, this is how it avoids needing to check the same unchanged files multiple times.) Here is the dangerous part: since the sync calls are limited to only the changed files, this means that everything else is "excluded", and would therefore be deleted if --delete-excluded is used. This is almost certainly not what you want -- and in fact, I would go so far as to say that Bisync should probably just force this flag off in a future version, for safety (there is a similar idea proposed for --exclude-if-present)

To accomplish what I think you want to do, what I'd recommend instead is to do this in two steps. First do your deletes, like:

rclone delete "dropbox:/New MyProject Team/test/season_1/job_1" --include "*.jpg"
rclone delete "/home/user/temp/MyProject/dropbox_editors/job_1_[42]" --include "*.jpg"

(notice the change from --exclude to --include there)

Then, do your bisync, without --delete-excluded:

rclone bisync "dropbox:/New MyProject Team/test/season_1/job_1" "/home/user/temp/MyProject/dropbox_editors/job_1_[42]" --create-empty-src-dirs --ignore-listing-checksum --exclude "*.jpg" --log-level DEBUG
1 Like

Thank you so much for the answer and explanation!

I thought that --delete-excluded flag should delete all excluded files only from the destination, what is the second path in bisync (I don't understand why I thought so). This was exactly what I needed: to delete the files only on a local machine but don't delete them on a server, like the selective sync in Dropbox.

So, yes, the solution with rclone delete <localpath> --include *.jpg before syncing is very nice in this case. Thank you so much again!

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.