How to specify what folders to sync and what to exclude -- include, exclude, filter?

[imthenachoman]

I have an arbitrary # of folders I want to backup to the cloud, and, for each folder I have specific things I want to exclude.

With rsync I am using using / as the source and then --files-from and --exclude-from:

--files-from

/home
/etc
/stuff

--exclude-from

/home/*/.tmp
/home/*/.cache
/stuff/lost+found

rsync
    --files-from="..."
    --exclude-from="..."
    /
    /backup/destination

I thought I could use the same flags with rclone but reading https://rclone.org/filtering/, I am not sure if --files-from and --exclude-from can be used together?

Would I use --files-from and --exclude-from just like I am with rsync:

rclone
    --files-from="..."
    --exclude-from="..."
    sync
        /
        remote:/

Or would I have to use --filter:

rclone
    --filter="+ /home"
    --filter="+ /etc"
    --filter="+ /stuff"
    --filter="- /home/*/.tmp"
    --filter="- /home/*/.cache"
    --filter="- /stuff/lost+found"
    sync
        /
        remote:/

Or is there a better way to do this?

[Animosity022]

I'd make a filter file and clean it up a little bit. You match top down, so you want to be most specific to least specific kind of like a firewall rule.

I think I did something similar to what you are asking but hard to be sure I got your rules translated in my brain right.

felix@gemini:~/test$ rclone ls /home/felix/test --filter-from filter
       84 filter
        0 home/test
        0 stuff/adfjad
        0 notme/aalkdfjaf
felix@gemini:~/test$ rclone ls /home/felix/test
       84 filter
        0 stuff/adfjad
        0 home/test
        0 notme/aalkdfjaf
        0 home/.tmp/blah
felix@gemini:~/test$ cat filter
- /home/.tmp/**
- /home/.cache/**
- /stuff/lost+found
+ home/**
+ etc/**
+ stuff/**

That gives you an idea. I always test with rclone ls and do a --dry-run after to ensure it will do what I expect and then actually run the copy or sync command.

[imthenachoman]

Thanks but I'm still not getting something.

If I run rlcone ls / I get thousands of files, all of the files on my system.

If I run rclone ls / --filter="+ /home/**" I would expect it to just include files in my /home but it is still returning all files in my system.

If I do a sync, I would need to do it from / because the files I want to sync are in different places on my system.

[Animosity022]

You'd want to add the exclude at the end in that example.

rclone ls / --filter='+ /home/**'  --filter='- *'

I find a file to be easier for me.

[imthenachoman]

Thanks. That solves me.

I am creating the filter list dynamically, in a script, and I don't want to create a temp file. I got this working:

INCLUDES=('/home/**' '/etc/**' '/stuff/**')
EXCLUDES=('/home/*/.gvfs/**' '/home/*/.cache/**' '/home/*/.local/share/Trash/**' '/home/*/.vscode-server/**' '/stuff/lost+found/**')

set -o noglob
        filterOptions=()
        for option in ${EXCLUDES[@]};
        do
            echo --filter="- $option" >> "$LOG_FILE"
            filterOptions+=(--filter="- $option")
        done

        for option in ${INCLUDES[@]};
        do
            echo --filter="- $option" >> "$LOG_FILE"
            filterOptions+=(--filter="+ $option")
        done
        
        echo --filter="- /**" >> "$LOG_FILE"
        filterOptions+=(--filter="- /**")
set +o noglob

/usr/bin/rclone                                            \
        --dry-run                                          \
        -v                                                 \
        --fast-list                                        \
        --skip-links                                       \
        --config="/root/.config/rclone/rclone.conf"        \
        "${filterOptions[@]}"                              \
        --backup-dir "remote:old"                          \
        --suffix ".$(/bin/date +\%Y\%m\%d\%H\%M\%S)"       \
        sync                                               \
            /                                              \
            "remote:new"                                   \
        >> "$LOG_FILE"                                     \

Thanks!