Sync two partitions with one rclone command?

wolfv · June 27, 2018, 6:07am

I want to back up two partitions on my home PC.
A DATA partition is mounted on the root partition.
On the file browser, it looks like one file system.

The bulk of files to be backed up are in the DATA partition.
There are also a few config files in the root partition’s home directory that I want to backup.

Is there a way to sync both partitions to a destination with one filter_rules file and one rclone command?
If so, should rclone sync use the --one-file-system option?
Or is it better to use two separate rclone commands?

Can the two partitions sync to one destination?
Or is it better to use a separate destination for each partition?

The following is what I tried so far.

Destination is a USB drive mounted on the root partition:

local:/run/media/wolfv/big_stick/wolfv_backup/last_snapshot

Here is the filter rules file, where “scripts” is a directory and “.spacemacs” is a file":

$ cat /DATA/Documents/test_filter_rules
+ DATA/scripts/**
+ home/wolfv/.spacemacs
- *

I expect rclone to “Not copying as --dry-run”, but it didn’t.
Rclone also unnecessarily looked a bunch of files and directories in /home/wolfv/.

Here is the directory structure:

$ ls /
bin  boot  DATA  dev  etc  home  lib  lib64  lost+found  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var
$ ls /DATA/scripts
clean_vim_swap  data_backup_for_linux_install.sh  gitstat  switch_sound_notes.org  switch_sound.sh  symlink_to_DATA.sh
$ ls /home/wolfv/.spacemacs
/home/wolfv/.spacemacs

Here is the rclone command:

$ rclone sync / local:/run/media/wolfv/big_stick/wolfv_backup/last_snapshot --filter-from=/DATA/Documents/test_filter_rules --one-file-system --log-level=DEBUG --dry-run
2018/06/27 04:29:34 DEBUG : rclone: Version "v1.41" starting with parameters ["rclone" "sync" "/" "local:/run/media/wolfv/big_stick/wolfv_backup/last_snapshot" "--filter-from=/DATA/Documents/test_filter_rules" "--one-file-system" "--log-level=DEBUG" "--dry-run"]
2018/06/27 04:29:34 DEBUG : Using config file from "/home/wolfv/.config/rclone/rclone.conf"
2018/06/27 04:29:34 INFO  : Local file system at /run/media/wolfv/big_stick/wolfv_backup/last_snapshot: Modify window is 1ns
2018/06/27 04:29:34 NOTICE: sbin: Can't follow symlink without -L/--copy-links
2018/06/27 04:29:34 NOTICE: bin: Can't follow symlink without -L/--copy-links
2018/06/27 04:29:34 NOTICE: lib64: Can't follow symlink without -L/--copy-links
2018/06/27 04:29:34 NOTICE: lib: Can't follow symlink without -L/--copy-links
2018/06/27 04:29:34 DEBUG : root: Excluded from sync (and deletion)
2018/06/27 04:29:34 DEBUG : usr: Excluded from sync (and deletion)
2018/06/27 04:29:34 DEBUG : lost+found: Excluded from sync (and deletion)
2018/06/27 04:29:34 DEBUG : srv: Excluded from sync (and deletion)
2018/06/27 04:29:34 DEBUG : mnt: Excluded from sync (and deletion)
2018/06/27 04:29:34 DEBUG : boot: Excluded from sync (and deletion)
2018/06/27 04:29:34 DEBUG : var: Excluded from sync (and deletion)
2018/06/27 04:29:34 DEBUG : opt: Excluded from sync (and deletion)
2018/06/27 04:29:34 DEBUG : etc: Excluded from sync (and deletion)
2018/06/27 04:29:34 NOTICE: home/wolfv/.vimrc: Can't follow symlink without -L/--copy-links
2018/06/27 04:29:34 NOTICE: home/wolfv/.gitignore_global: Can't follow symlink without -L/--copy-links
2018/06/27 04:29:34 NOTICE: home/wolfv/scripts: Can't follow symlink without -L/--copy-links
2018/06/27 04:29:34 NOTICE: home/wolfv/Documents: Can't follow symlink without -L/--copy-links
2018/06/27 04:29:34 NOTICE: home/wolfv/.fonts: Can't follow symlink without -L/--copy-links
2018/06/27 04:29:34 NOTICE: home/wolfv/.hunspell_en_US: Can't follow symlink without -L/--copy-links
2018/06/27 04:29:34 NOTICE: home/wolfv/cust_configs: Can't follow symlink without -L/--copy-links
2018/06/27 04:29:34 NOTICE: home/wolfv/Pictures: Can't follow symlink without -L/--copy-links
2018/06/27 04:29:34 NOTICE: home/wolfv/.gitconfig: Can't follow symlink without -L/--copy-links
2018/06/27 04:29:34 NOTICE: home/wolfv/.git-credentials: Can't follow symlink without -L/--copy-links
2018/06/27 04:29:34 DEBUG : home/wolfv/Desktop: Excluded from sync (and deletion)
2018/06/27 04:29:34 DEBUG : home/wolfv/.bash_history: Excluded from sync (and deletion)
2018/06/27 04:29:34 DEBUG : home/wolfv/.bash_logout: Excluded from sync (and deletion)
2018/06/27 04:29:34 DEBUG : home/wolfv/Recordings: Excluded from sync (and deletion)
2018/06/27 04:29:34 DEBUG : home/wolfv/.bashrc: Excluded from sync (and deletion)
2018/06/27 04:29:34 DEBUG : home/wolfv/.hunspell_en_US_orig: Excluded from sync (and deletion)
2018/06/27 04:29:34 DEBUG : home/wolfv/test_rclone_data: Excluded from sync (and deletion)
2018/06/27 04:29:34 DEBUG : home/wolfv/Downloads: Excluded from sync (and deletion)
2018/06/27 04:29:34 DEBUG : home/wolfv/Public: Excluded from sync (and deletion)
2018/06/27 04:29:34 DEBUG : home/wolfv/.lesshst: Excluded from sync (and deletion)
2018/06/27 04:29:34 DEBUG : home/wolfv/.cache: Excluded from sync (and deletion)
2018/06/27 04:29:34 DEBUG : home/wolfv/rclone_jobber_test: Excluded from sync (and deletion)
2018/06/27 04:29:34 DEBUG : home/wolfv/.esd_auth: Excluded from sync (and deletion)
2018/06/27 04:29:34 DEBUG : home/wolfv/rclone.conf: Excluded from sync (and deletion)
2018/06/27 04:29:34 DEBUG : home/wolfv/.ccache: Excluded from sync (and deletion)
2018/06/27 04:29:34 DEBUG : home/wolfv/.local: Excluded from sync (and deletion)
2018/06/27 04:29:34 DEBUG : home/wolfv/.emacs.d: Excluded from sync (and deletion)
2018/06/27 04:29:34 DEBUG : home/wolfv/.ICEauthority: Excluded from sync (and deletion)
2018/06/27 04:29:34 DEBUG : home/wolfv/.gnupg: Excluded from sync (and deletion)
2018/06/27 04:29:34 DEBUG : home/wolfv/.viminfo: Excluded from sync (and deletion)
2018/06/27 04:29:34 DEBUG : home/wolfv/.spacemacs_orig: Excluded from sync (and deletion)
2018/06/27 04:29:34 DEBUG : home/wolfv/.emacs.d.bak: Excluded from sync (and deletion)
2018/06/27 04:29:34 DEBUG : home/wolfv/.config: Excluded from sync (and deletion)
2018/06/27 04:29:34 DEBUG : home/wolfv/.pki: Excluded from sync (and deletion)
2018/06/27 04:29:34 DEBUG : home/wolfv/.mozilla: Excluded from sync (and deletion)
2018/06/27 04:29:34 DEBUG : home/wolfv/.bash_profile: Excluded from sync (and deletion)
2018/06/27 04:29:34 NOTICE: home/wolfv/.spacemacs: Not copying as --dry-run
2018/06/27 04:29:34 INFO  : Local file system at /run/media/wolfv/big_stick/wolfv_backup/last_snapshot: Waiting for checks to finish
2018/06/27 04:29:34 INFO  : Local file system at /run/media/wolfv/big_stick/wolfv_backup/last_snapshot: Waiting for transfers to finish
2018/06/27 04:29:34 INFO  : Waiting for deletions to finish
2018/06/27 04:29:34 INFO  : 
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:                 0
Transferred:            1
Elapsed time:          0s

2018/06/27 04:29:34 DEBUG : 2 go routines active
2018/06/27 04:29:34 DEBUG : rclone: Version "v1.41" finishing with parameters ["rclone" "sync" "/" "local:/run/media/wolfv/big_stick/wolfv_backup/last_snapshot" "--filter-from=/DATA/Documents/test_filter_rules" "--one-file-system" "--log-level=DEBUG" "--dry-run"]

ncw · June 27, 2018, 8:38am

Do you have the comments in your filter file? The filter file doesn’t support comments on the end of lines as you might have # in your file name.

I also think you need leading /, so and a trailing ** on the spacemacs

$ cat /DATA/Documents/test_filter_rules
+ DATA/scripts/**
+ home/wolfv/.spacemacs/**
- *

Otherwise I think your approach should work and be flexible enough to add more destinations.

wolfv · June 27, 2018, 11:07am

Hi ncw. Thanks for the information.

“scripts” is a directory and “.spacemacs” is a file.
I removed the comments from filter rules, and ran it again with similar results (I copy & pasted the new results into the original post).

Something is still amiss.
There are no “Not copying as --dry-run”.
And rclone unnecessarily looked a bunch of files and directories in /home/wolfv/.

ncw · June 27, 2018, 1:39pm

It does now talk about .spacemacs

Note that you still want the leading / to the paths otherwise rclone has to do more work searching the whole filesystem for any directories called DATA/scripts/ not just in the root.

Is /DATA on a different filesystem maybe? If so you’ll need to lose the --one-file-system flag.

wolfv · June 27, 2018, 2:13pm

So glad you found the .spacemacs file.

That just leaves one more mystery. Why does rclone not find /DATA/scripts/** ?
I suspected it may have something to do with /DATA being a separate partition, mounted to root.

Filter rules file:

$ cat /DATA/Documents/test_filter_rules
+ /DATA/scripts/**
- *

/DATA/scripts/ contains several files:

$ ls /DATA/scripts/
clean_vim_swap  data_backup_for_linux_install.sh  gitstat  switch_sound_notes.org  switch_sound.sh  symlink_to_DATA.sh

The rclone command does not find /DATA/scripts/:

$ rclone sync / local:/run/media/wolfv/big_stick/wolfv_backup/last_snapshot --filter-from=/DATA/Documents/test_filter_rules --one-file-system --log-level=DEBUG --dry-run
2018/06/27 13:15:29 DEBUG : rclone: Version "v1.41" starting with parameters ["rclone" "sync" "/" "local:/run/media/wolfv/big_stick/wolfv_backup/last_snapshot" "--filter-from=/DATA/Documents/test_filter_rules" "--one-file-system" "--log-level=DEBUG" "--dry-run"]
2018/06/27 13:15:29 DEBUG : Using config file from "/home/wolfv/.config/rclone/rclone.conf"
2018/06/27 13:15:29 INFO  : Local file system at /run/media/wolfv/big_stick/wolfv_backup/last_snapshot: Modify window is 1ns
2018/06/27 13:15:29 NOTICE: sbin: Can't follow symlink without -L/--copy-links
2018/06/27 13:15:29 NOTICE: bin: Can't follow symlink without -L/--copy-links
2018/06/27 13:15:29 NOTICE: lib64: Can't follow symlink without -L/--copy-links
2018/06/27 13:15:29 NOTICE: lib: Can't follow symlink without -L/--copy-links
2018/06/27 13:15:29 DEBUG : home: Excluded from sync (and deletion)
2018/06/27 13:15:29 DEBUG : root: Excluded from sync (and deletion)
2018/06/27 13:15:29 DEBUG : usr: Excluded from sync (and deletion)
2018/06/27 13:15:29 DEBUG : lost+found: Excluded from sync (and deletion)
2018/06/27 13:15:29 DEBUG : srv: Excluded from sync (and deletion)
2018/06/27 13:15:29 DEBUG : mnt: Excluded from sync (and deletion)
2018/06/27 13:15:29 DEBUG : boot: Excluded from sync (and deletion)
2018/06/27 13:15:29 DEBUG : var: Excluded from sync (and deletion)
2018/06/27 13:15:29 DEBUG : opt: Excluded from sync (and deletion)
2018/06/27 13:15:29 DEBUG : etc: Excluded from sync (and deletion)
2018/06/27 13:15:29 INFO  : Local file system at /run/media/wolfv/big_stick/wolfv_backup/last_snapshot: Waiting for checks to finish
2018/06/27 13:15:29 INFO  : Local file system at /run/media/wolfv/big_stick/wolfv_backup/last_snapshot: Waiting for transfers to finish
2018/06/27 13:15:29 INFO  : Waiting for deletions to finish
2018/06/27 13:15:29 INFO  : 
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:                 0
Transferred:            0
Elapsed time:          0s

2018/06/27 13:15:29 DEBUG : 3 go routines active
2018/06/27 13:15:29 DEBUG : rclone: Version "v1.41" finishing with parameters ["rclone" "sync" "/" "local:/run/media/wolfv/big_stick/wolfv_backup/last_snapshot" "--filter-from=/DATA/Documents/test_filter_rules" "--one-file-system" "--log-level=DEBUG" "--dry-run"]

ncw · June 28, 2018, 9:05am

Yes, that is what I meant above. Try without the --one-file-system flag and it should find it hopefully!

wolfv · June 28, 2018, 4:31pm

Yes, that was it. Removing the --one-file-system flag fixed it.
Thanks for your help. You’re the best.
The following is a summary of what I did.

Source is / (root). /DATA is on a different partition mounted on root.
Destination is USB drive mounted on root:

/run/media/wolfv/big_stick/wolfv_backup_test

I created a local remote named “local”, and put the USB drive on it:

local:/run/media/wolfv/big_stick/wolfv_backup_test

This rclone command successfully sync without the --one-file-system flag:

$ rclone sync / local:/run/media/wolfv/big_stick/wolfv_backup_test --filter-from=/DATA/Documents/test_filter_rules --log-level=DEBUG
2018/06/28 05:16:06 DEBUG : rclone: Version "v1.41" starting with parameters ["rclone" "sync" "/" "local:/run/media/wolfv/big_stick/wolfv_backup_test" "--filter-from=/DATA/Documents/test_filter_rules" "--log-level=DEBUG"]
2018/06/28 05:16:06 DEBUG : Using config file from "/home/wolfv/.config/rclone/rclone.conf"
2018/06/28 05:16:06 INFO  : Local file system at /run/media/wolfv/big_stick/wolfv_backup_test: Modify window is 1ns
2018/06/28 05:16:06 NOTICE: sbin: Can't follow symlink without -L/--copy-links
2018/06/28 05:16:06 NOTICE: bin: Can't follow symlink without -L/--copy-links
2018/06/28 05:16:06 NOTICE: lib64: Can't follow symlink without -L/--copy-links
2018/06/28 05:16:06 NOTICE: lib: Can't follow symlink without -L/--copy-links
2018/06/28 05:16:06 DEBUG : home: Excluded from sync (and deletion)
2018/06/28 05:16:06 DEBUG : dev: Excluded from sync (and deletion)
2018/06/28 05:16:06 DEBUG : tmp: Excluded from sync (and deletion)
2018/06/28 05:16:06 DEBUG : root: Excluded from sync (and deletion)
2018/06/28 05:16:06 DEBUG : usr: Excluded from sync (and deletion)
2018/06/28 05:16:06 DEBUG : lost+found: Excluded from sync (and deletion)
2018/06/28 05:16:06 DEBUG : srv: Excluded from sync (and deletion)
2018/06/28 05:16:06 DEBUG : mnt: Excluded from sync (and deletion)
2018/06/28 05:16:06 DEBUG : boot: Excluded from sync (and deletion)
2018/06/28 05:16:06 DEBUG : run: Excluded from sync (and deletion)
2018/06/28 05:16:06 DEBUG : proc: Excluded from sync (and deletion)
2018/06/28 05:16:06 DEBUG : var: Excluded from sync (and deletion)
2018/06/28 05:16:06 DEBUG : sys: Excluded from sync (and deletion)
2018/06/28 05:16:06 DEBUG : opt: Excluded from sync (and deletion)
2018/06/28 05:16:06 DEBUG : etc: Excluded from sync (and deletion)
2018/06/28 05:16:06 DEBUG : DATA/lost+found: Excluded from sync (and deletion)
2018/06/28 05:16:06 DEBUG : DATA/.local: Excluded from sync (and deletion)
2018/06/28 05:16:06 DEBUG : DATA/cust_configs: Excluded from sync (and deletion)
2018/06/28 05:16:06 DEBUG : DATA/Pictures: Excluded from sync (and deletion)
2018/06/28 05:16:06 DEBUG : DATA/Documents: Excluded from sync (and deletion)
2018/06/28 05:16:06 DEBUG : DATA/.fonts: Excluded from sync (and deletion)
2018/06/28 05:16:06 DEBUG : DATA/.Trash-1000: Excluded from sync (and deletion)
2018/06/28 05:16:06 INFO  : DATA/scripts/data_backup_for_linux_install.sh: Copied (new)
2018/06/28 05:16:06 INFO  : DATA/scripts/gitstat: Copied (new)
2018/06/28 05:16:06 INFO  : DATA/scripts/switch_sound.sh: Copied (new)
2018/06/28 05:16:06 INFO  : Local file system at /run/media/wolfv/big_stick/wolfv_backup_test: Waiting for checks to finish
2018/06/28 05:16:06 INFO  : Local file system at /run/media/wolfv/big_stick/wolfv_backup_test: Waiting for transfers to finish
2018/06/28 05:16:06 INFO  : DATA/scripts/switch_sound_notes.org: Copied (new)
2018/06/28 05:16:06 INFO  : DATA/scripts/symlink_to_DATA.sh: Copied (new)
2018/06/28 05:16:06 INFO  : DATA/scripts/clean_vim_swap: Copied (new)
2018/06/28 05:16:06 INFO  : Waiting for deletions to finish
2018/06/28 05:16:06 INFO  : 
Transferred:   13.242 kBytes (706.045 kBytes/s)
Errors:                 0
Checks:                 0
Transferred:            6
Elapsed time:          0s

2018/06/28 05:16:06 DEBUG : 3 go routines active
2018/06/28 05:16:06 DEBUG : rclone: Version "v1.41" finishing with parameters ["rclone" "sync" "/" "local:/run/media/wolfv/big_stick/wolfv_backup_test" "--filter-from=/DATA/Documents/test_filter_rules" "--log-level=DEBUG"]

ncw · June 28, 2018, 9:06pm

Great! It was my fault - I told you to put it there in the first place - sorry!

I’m glad you got it working and thanks for summarizing what you did.