Need help to understand filtering commands

xilatos · February 9, 2021, 11:01pm

What is the problem you are having with rclone?

I'm having a hard time understanding how filtering works in rclone.

What is your rclone version (output from `rclone version`)

v1.54.0

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Windows 10 LTSC 2019 (x64).

Which cloud storage system are you using? (eg Google Drive)

Local storage.

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

I want to sync up two folders locally. I want to exclude two subdirectories from the synchronization (excl1 and excl2).

I did some testing. It looks like this:
F:
├── filter.txt
├── bkp
├── res
│ ├── pic1.png
│ ├── excl1
│ │ ├── PLA.jpg
│ │ └── ABS.jpg
│ ├── excl2
│ │ └── document.pdf

Since it is a test I decided to sync \res\ with \bkp\ to keep the commands simple.

From the filter commands explained in https://rclone.org/filtering/ ,
I tried the following:
· rclone sync F:\res\ F:\bkp\ --exclude f:/res/excl1/ --exclude f:/res/excl2/
· rclone sync F:\res\ F:\bkp\ --exclude /res/excl1/ --exclude /res/excl2/
None of these commands excluded the folders.

· rclone copy F:\res\ F:\bkp\ --exclude f:/res/excl1/ --exclude f:/res/excl2/
· rclone copy F:\res\ F:\bkp\ --exclude /res/excl1/ --exclude /res/excl2/
Nothing was copied.

· rclone sync F:\res\ F:\bkp\ --filter-from F:\filter.txt
With filter.txt containing:
- f:/res/excl1
- f:/res/excl2
The folders were not excluded.

· rclone sync F:\res\ F:\bkp\ --filter-from F:\filter.txt
With filter.txt containing:
- /res/excl1
- /res/excl2
The folders were not excluded.

· rclone sync F:\res\ F:\bkp\ --filter-from F:\filter.txt
With filter.txt containing:
- f:/res/excl1
- f:/res/excl2
- **
Nothing was copied.

I tried excluding files with:
· rclone sync F:\res\ F:\bkp\ --exclude /F:/res/excl1/abs.jpg --ignore-case
· rclone sync F:\res\ F:\bkp\ --exclude /F:/res/excl2/document.pdf --ignore-case
It worked perfectly.

I don't know how to exclude directories. It is definetely because I did not understand the function of the filtering commands correctly (https://rclone.org/filtering/). I would highly appreciate any help to understand them.

The rclone config contents with secrets removed.

[TOSHIBA]
type = memory

A log from the command with the `-vv` flag

C:\users\xilatos> rclone sync F:\res\ F:\bkp\ --exclude /res/excl1/ --exclude /res/excl2/ -vv
2021/02/09 23:57:41 DEBUG : rclone: Version "v1.54.0" starting with parameters ["C:\\ProgramData\\chocolatey\\lib\\rclone.portable\\tools\\rclone-v1.54.0-windows-amd64\\rclone.exe" "sync" "F:\\res\\" "F:\\bkp\\" "--exclude" "/res/excl1/" "--exclude" "/res/excl2/" "-vv"]
2021/02/09 23:57:41 DEBUG : Creating backend with remote "F:\\res\\"
2021/02/09 23:57:41 DEBUG : Using config file from "C:\\Users\\xilatos\\.config\\rclone\\rclone.conf"
2021/02/09 23:57:41 DEBUG : fs cache: renaming cache item "F:\\res\\" to be canonical "//?/F:/res/"
2021/02/09 23:57:41 DEBUG : Creating backend with remote "F:\\bkp\\"
2021/02/09 23:57:41 DEBUG : fs cache: renaming cache item "F:\\bkp\\" to be canonical "//?/F:/bkp/"
2021/02/09 23:57:41 DEBUG : Local file system at //?/F:/bkp/: Waiting for checks to finish
2021/02/09 23:57:41 DEBUG : Local file system at //?/F:/bkp/: Waiting for transfers to finish
2021/02/09 23:57:44 DEBUG : pic1.png: MD5 = 28b22af79c4dd8dba139b38f48d2ea33 OK
2021/02/09 23:57:44 INFO  : pic1.png: Copied (new)
2021/02/09 23:57:44 DEBUG : excl2/document.pdf: MD5 = 047afcd69d190f528e7e8012f3dbbd72 OK
2021/02/09 23:57:44 DEBUG : excl1/PLA.jpg: MD5 = 3a20b95c4dcf37371613e30c06247c6f OK
2021/02/09 23:57:44 INFO  : excl2/document.pdf: Copied (new)
2021/02/09 23:57:44 INFO  : excl1/PLA.jpg: Copied (new)
2021/02/09 23:57:44 DEBUG : excl1/ABS.jpg: MD5 = afbea17255d90e94a6b1f70119ecfba0 OK
2021/02/09 23:57:44 INFO  : excl1/ABS.jpg: Copied (new)
2021/02/09 23:57:44 DEBUG : Waiting for deletions to finish
2021/02/09 23:57:44 INFO  :
Transferred:       10.257M / 10.257 MBytes, 100%, 2.807 MBytes/s, ETA 0s
Transferred:            4 / 4, 100%
Elapsed time:         3.9s

(filter.txt includes "- f:/res/excl1" and "- f:/res/excl2")
C:\users\xilatos> rclone sync F:\res\ F:\bkp\ --filter-from f:\filter.txt -vv
2021/02/09 23:46:12 DEBUG : rclone: Version "v1.54.0" starting with parameters ["C:\\ProgramData\\chocolatey\\lib\\rclone.portable\\tools\\rclone-v1.54.0-windows-amd64\\rclone.exe" "sync" "F:\\res\\" "F:\\bkp\\" "--filter-from" "f:\\filter.txt" "-vv"]
2021/02/09 23:46:12 DEBUG : Creating backend with remote "F:\\res\\"
2021/02/09 23:46:12 DEBUG : Using config file from "C:\\Users\\xilatos\\.config\\rclone\\rclone.conf"
2021/02/09 23:46:12 DEBUG : fs cache: renaming cache item "F:\\res\\" to be canonical "//?/F:/res/"
2021/02/09 23:46:12 DEBUG : Creating backend with remote "F:\\bkp\\"
2021/02/09 23:46:12 DEBUG : fs cache: renaming cache item "F:\\bkp\\" to be canonical "//?/F:/bkp/"
2021/02/09 23:46:12 DEBUG : Local file system at //?/F:/bkp/: Waiting for checks to finish
2021/02/09 23:46:12 DEBUG : Local file system at //?/F:/bkp/: Waiting for transfers to finish
2021/02/09 23:46:12 DEBUG : pic1.png: MD5 = 28b22af79c4dd8dba139b38f48d2ea33 OK
2021/02/09 23:46:12 INFO  : pic1.png: Copied (new)
2021/02/09 23:46:12 DEBUG : excl1/PLA.jpg: MD5 = 3a20b95c4dcf37371613e30c06247c6f OK
2021/02/09 23:46:12 INFO  : excl1/PLA.jpg: Copied (new)
2021/02/09 23:46:12 DEBUG : excl2/document.pdf: MD5 = 047afcd69d190f528e7e8012f3dbbd72 OK
2021/02/09 23:46:12 INFO  : excl2/document.pdf: Copied (new)
2021/02/09 23:46:13 DEBUG : excl1/ABS.jpg: MD5 = afbea17255d90e94a6b1f70119ecfba0 OK
2021/02/09 23:46:13 INFO  : excl1/ABS.jpg: Copied (new)
2021/02/09 23:46:13 DEBUG : Waiting for deletions to finish
2021/02/09 23:46:13 INFO  :
Transferred:       10.257M / 10.257 MBytes, 100%, 18.796 MBytes/s, ETA 0s
Transferred:            4 / 4, 100%
Elapsed time:         0.7s

(filter.txt includes "- f:/res/excl1", "- f:/res/excl2" and " -** ")
PS C:\users\xilatos> rclone sync F:\res\ F:\bkp\ --filter-from f:\filter.txt -vv
2021/02/09 23:50:48 DEBUG : rclone: Version "v1.54.0" starting with parameters ["C:\\ProgramData\\chocolatey\\lib\\rclone.portable\\tools\\rclone-v1.54.0-windows-amd64\\rclone.exe" "sync" "F:\\res\\" "F:\\bkp\\" "--filter-from" "f:\\filter.txt" "-vv"]
2021/02/09 23:50:48 DEBUG : Creating backend with remote "F:\\res\\"
2021/02/09 23:50:48 DEBUG : Using config file from "C:\\Users\\xilatos\\.config\\rclone\\rclone.conf"
2021/02/09 23:50:48 DEBUG : fs cache: renaming cache item "F:\\res\\" to be canonical "//?/F:/res/"
2021/02/09 23:50:48 DEBUG : Creating backend with remote "F:\\bkp\\"
2021/02/09 23:50:48 DEBUG : fs cache: renaming cache item "F:\\bkp\\" to be canonical "//?/F:/bkp/"
2021/02/09 23:50:48 DEBUG : excl1: Excluded
2021/02/09 23:50:48 DEBUG : excl2: Excluded
2021/02/09 23:50:48 DEBUG : pic1.png: Excluded
2021/02/09 23:50:48 DEBUG : Local file system at //?/F:/bkp/: Waiting for checks to finish
2021/02/09 23:50:48 DEBUG : Local file system at //?/F:/bkp/: Waiting for transfers to finish
2021/02/09 23:50:48 DEBUG : Waiting for deletions to finish
2021/02/09 23:50:48 INFO  : There was nothing to transfer
2021/02/09 23:50:48 INFO  :
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Elapsed time:         0.2s

asdffdsa · February 9, 2021, 11:33pm

hello and welcome to the forum,

no worries, filters are very confusing and sometimes there is more than one way to do it.

the filter is relative to the root of the source, in this case the root is f:\res\

so i think this will work but be careful with sync, as it can and will delete files in the dest. best to use --dry-run or test with rclone ls, not rclone sync

rclone sync f:\res\ F:\bkp\ --exclude=/excl1/** --exclude=/excl2/** --dry-run

xilatos · February 10, 2021, 3:15am

Thank you very much for your comment. I tried the command you provided and it worked like a charm! I will test with ls instead of sync from now on.

Edward_Barker · February 10, 2021, 12:08pm

@asdffdsa, I have borrowed a variant of the answer as a pr to add an example to the docs filtering page.

https://github.com/rclone/rclone/pull/5010

system · February 13, 2021, 12:08pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.