Sync is not deleting folders at the remote when a previously include rule folder is no longer used

racofer · July 7, 2020, 10:13pm

What is the problem you are having with rclone?

Although I am not listing the real scenario where I have encountered this issue due to it containing sensitive data, the following test scenario shows the same behavior. I believe it is something wrong with my filtering rules. Say I have the following folder structure:

Test
├── Folder 1
│   ├── a.txt
│   ├── b.txt
│   └── c.txt
├── Folder 2
│   ├── Dont Sync This
│   │   └── dont.txt
│   ├── a.txt
│   ├── b.txt
│   └── c.txt
├── Folder 3
│   ├── Dont Sync This
│   │   └── dont.txt
│   ├── Hello
│   │   ├── Crap
│   │   │   ├── a.bmp
│   │   │   ├── b.bmp
│   │   │   └── c.bmp
│   │   ├── a.jpg
│   │   ├── b.jpg
│   │   └── c.jpg
│   ├── Stuff
│   │   ├── a.txt
│   │   ├── b.txt
│   │   └── c.txt
│   ├── a.txt
│   ├── b.txt
│   └── c.txt
├── a.py
├── b.bat
└── c.bat

I want to sync all files in it with the exception of some folders (clearly shown as "Dont Sync This") to a remote. I can sync it perfectly fine with the following command (testing on the local file system, however, the behavior is the same with Google Drive or OneDrive):

rclone -vv --include="/*" --include="/Folder 1/*.txt" --include="/Folder 2/*.txt" --include="/Folder 3/*.txt" --include="/Folder 3/Hello/**" --include="/Folder 3/Stuff/**" sync V:/test V:/test2

After that, everything syncs just fine. Deleted files get deleted, new files get added, etc.

Now say I no longer want to sync the "Hello" folder from above. It still exists at the source, however, it should no longer be synced. I have removed it from the include filter as:

rclone -vv --include="/*" --include="/Folder 1/*.txt" --include="/Folder 2/*.txt" --include="/Folder 3/*.txt" --include="/Folder 3/Stuff/**" sync V:/test V:/test2

And it is no longer marked for sync, however, it is also not deleted from the remote. I have tried using filter instead of include and added an exclusion filter to everything else, but the remote files are not marked for deletion:

rclone -vv --filter="+ /*" --filter="+ /Folder 1/*.txt" --filter="+ /Folder 2/*.txt" --filter="+ /Folder 3/*.txt" --filter="+ /Folder 3/Stuff/**" --filter="- *" sync V:/test v:/test2

And the Debug shows that the "Hello" folder is now excluded:

2020/07/07 18:38:27 DEBUG : Folder 3/Hello: Excluded

How do I get the updated sync command to delete the excluded files from the remote? What am I doing wrong here?

What is your rclone version (output from `rclone version`)

rclone v1.52.2

os/arch: windows/amd64

go version: go1.14.4

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Windows 10 64 bit

Which cloud storage system are you using? (eg Google Drive)

Google Drive, OneDrive and Local File System with/without Crypt on either of them.

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

See above.

The rclone config contents with secrets removed.

Not needed as testing with the local file system shows the same issue.

A log from the command with the `-vv` flag

2020/07/07 18:44:57 DEBUG : rclone: Version "v1.52.2" starting with parameters ["rclone" "-vv" "--filter=+ /Folder 1/*.txt" "--filter=+ /Folder 2/*.txt" "--filter=+ /Folder 3/*.txt" "--filter=+ /Folder 3/Stuff/**" "--filter=- **" "sync" "V:/test" "v:/test2"]
2020/07/07 18:45:00 DEBUG : Using config file from "C:\\Users\\XXX\\.config\\rclone\\rclone.conf"
2020/07/07 18:45:00 DEBUG : fs cache: renaming cache item "V:/test" to be canonical "//?/V:/test"
2020/07/07 18:45:00 DEBUG : fs cache: renaming cache item "v:/test2" to be canonical "//?/v:/test2"
2020/07/07 18:45:00 DEBUG : a.py: Excluded
2020/07/07 18:45:00 DEBUG : b.bat: Excluded
2020/07/07 18:45:00 DEBUG : c.bat: Excluded
2020/07/07 18:45:00 DEBUG : a.py: Excluded
2020/07/07 18:45:00 DEBUG : b.bat: Excluded
2020/07/07 18:45:00 DEBUG : c.bat: Excluded
2020/07/07 18:45:00 DEBUG : Folder 3/Hello: Excluded
2020/07/07 18:45:00 DEBUG : Folder 3/Dont Sync This: Excluded
2020/07/07 18:45:00 DEBUG : Folder 1/a.txt: Size and modification time the same (differ by 0s, within tolerance 100ns)
2020/07/07 18:45:00 DEBUG : Folder 1/a.txt: Unchanged skipping
2020/07/07 18:45:00 DEBUG : Folder 3/Hello: Excluded
2020/07/07 18:45:00 DEBUG : Folder 3/a.txt: Size and modification time the same (differ by 0s, within tolerance 100ns)
2020/07/07 18:45:00 DEBUG : Folder 3/a.txt: Unchanged skipping
2020/07/07 18:45:00 DEBUG : Folder 3/b.txt: Size and modification time the same (differ by 0s, within tolerance 100ns)
2020/07/07 18:45:00 DEBUG : Folder 3/b.txt: Unchanged skipping
2020/07/07 18:45:00 DEBUG : Folder 1/c.txt: Size and modification time the same (differ by 0s, within tolerance 100ns)
2020/07/07 18:45:00 DEBUG : Folder 1/c.txt: Unchanged skipping
2020/07/07 18:45:00 DEBUG : Folder 1/b.txt: Size and modification time the same (differ by 0s, within tolerance 100ns)
2020/07/07 18:45:00 DEBUG : Folder 1/b.txt: Unchanged skipping
2020/07/07 18:45:00 DEBUG : Folder 2/Dont Sync This: Excluded
2020/07/07 18:45:00 DEBUG : Folder 3/c.txt: Size and modification time the same (differ by 0s, within tolerance 100ns)
2020/07/07 18:45:00 DEBUG : Folder 3/c.txt: Unchanged skipping
2020/07/07 18:45:00 DEBUG : Folder 3/Stuff/a.txt: Size and modification time the same (differ by 0s, within tolerance 100ns)
2020/07/07 18:45:00 DEBUG : Folder 3/Stuff/a.txt: Unchanged skipping
2020/07/07 18:45:00 DEBUG : Folder 3/Stuff/b.txt: Size and modification time the same (differ by 0s, within tolerance 100ns)
2020/07/07 18:45:00 DEBUG : Folder 3/Stuff/c.txt: Size and modification time the same (differ by 0s, within tolerance 100ns)
2020/07/07 18:45:00 DEBUG : Folder 2/a.txt: Size and modification time the same (differ by 0s, within tolerance 100ns)
2020/07/07 18:45:00 DEBUG : Folder 2/b.txt: Size and modification time the same (differ by 0s, within tolerance 100ns)
2020/07/07 18:45:00 DEBUG : Folder 3/Stuff/c.txt: Unchanged skipping
2020/07/07 18:45:00 DEBUG : Local file system at //?/v:/test2: Waiting for checks to finish
2020/07/07 18:45:00 DEBUG : Folder 3/Stuff/b.txt: Unchanged skipping
2020/07/07 18:45:00 DEBUG : Folder 2/b.txt: Unchanged skipping
2020/07/07 18:45:00 DEBUG : Folder 2/c.txt: Size and modification time the same (differ by 0s, within tolerance 100ns)
2020/07/07 18:45:00 DEBUG : Folder 2/c.txt: Unchanged skipping
2020/07/07 18:45:00 DEBUG : Folder 2/a.txt: Unchanged skipping
2020/07/07 18:45:00 DEBUG : Local file system at //?/v:/test2: Waiting for transfers to finish
2020/07/07 18:45:00 DEBUG : Waiting for deletions to finish
2020/07/07 18:45:00 INFO  : There was nothing to transfer
2020/07/07 18:45:00 INFO  :
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:                12 / 12, 100%
Elapsed time:         0.0s

2020/07/07 18:45:00 DEBUG : 2 go routines active

racofer · July 7, 2020, 10:35pm

Ok, so after hours looking at this, now I stumble upon the --delete-excluded flag which does what I want. Is this the proper way to handle my situation, or could it be managed some other way?

ncw · July 8, 2020, 11:07am

This is the correct flag to use. Note that --delete-excluded is a dangerous flag that can lead to data loss - get your filters wrong and rclone will delete loads of stuff on the destination. It is fine to use it on a sync that is working properly though - it just isn't the default.

I always set syncs up like this

try first with rclone copy
when that is working then use rclone sync
and when that is working (and only if necessary) use rclone sync --delete-excluded

racofer · July 8, 2020, 4:25pm

Got it, thanks!

asdffdsa · July 8, 2020, 4:30pm

let's be just a little more paranoid...

try first with rclone copy
when that is working then use rclone sync
when that is working then use rclone sync --delete-excluded --dry-run -v and read the output
and when that is working (and only if necessary) use rclone sync source: dest: --delete-excluded --backup-dir=dest:path/to/backupdir

system · July 11, 2020, 4:30pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.