I've made a nice backup script tutorial for rclone

Howto Guides
1

When I wrote my rclone backup script, I didn’t find many example rclone scripts.
So I thought somebody else could use the script, or modify it to their own needs.

I use the rclone script to backup my home PC to local and remote storage, and it really works well.
Find more information here: https://github.com/wolfv6/rclone_jobber

A few features:

  • Abort if job is already running (maybe previous run didn’t finish)
  • Pop-up for error conditions
  • Logging
  • Tutorial
  • Backup-job and restore-job examples

Any suggestions to improve on the script would be welcome.

Rclone was a pleasure to work with; the commands are nicely structured and the documentation is clear.
A big thank you to the writer of rclone.

6 Likes
2

Very nice :slight_smile: Do you want to add a link to it in the the rclone wiki - maybe in the 3rd party integrations section?

1 Like
3

Thanks for the invite; I added a link to 3rd Party Integrations (after RcloneBrowser).

1 Like
4

Hi wolfv, Thanks for sharing. I wasn’t aware of the --backup-dir option in rclone!

A couple of months ago I needed something similar and end up creating https://github.com/bcardiff/docker-rclone It was some features I usually like to see in a backup cron like time window and status check.

That docker image and others are listed in https://github.com/ncw/rclone/wiki/Docker-images .

1 Like
5

You’re welcome. And thanks for pointing me to your backup script.
It inspired me to add range checks for null strings, empty source, and a cron monitoring service.

rclone_jobber version 1.1 is released.

1 Like
6

Hi @wolfv,

I’ve been experimenting with rclone_jobber and I really like it!

By default it saves to a ‘latest_backup’ folder, which I see is “$new” in the script. If I already have a substantial amount of files that I used my own rclone script to back up to a destination (in this case a B2 bucket) in a folder called, say, “pictures”. I don’t want to back everything up again - I was hoping I could point my jobber script at the existing backup and just run periodic sync scripts. The obvious way would be to just change the $new variable in the rclone_jobber script, but I’d like to use that script to run separate scripts for different content types.

Short of my going in and hacking at rclone_jobber.sh, do you have any other suggestions how to manage this?

7

Hi @joti ,

As is, rclone_jobber.sh will always backup to “last_snapshot”.
From rclone_jobber.sh:

dest="$2"
...
new="last_snapshot"
...
cmd="rclone sync $source $dest/$new $backup_dir $log_option $options"
//$dest is also used in $backup_dir

Can you rearrange your existing destination backup directories to match the directory structure used by rclone_jobber? i.e. put them in “last_snap_shot”.
rclone_jobber directory structures are illustrated in https://github.com/wolfv6/rclone_jobber/blob/master/rclone_jobber_tutorial.org#3-move_old_files_to

If you can not rearrange your existing destination backup directories, you can customize the rclone_jobber.sh script.
A minimal rclone_jobber.sh customization would be to move the $new variable to a parameter, for example:

new="$7"

And then specify “$new” in the backup jobs.
Write a separate backup job for each directory e.g. “job_backup_pictures” backs up to new=“pictures”.
Example job_backup_pictures:

#!/usr/bin/env sh

source="$HOME/test_rclone_data"
dest="$usb/test_rclone_backup"
move_old_files_to="dated_files"
options="--filter-from=$rclone_jobber/examples/filter_rules --checksum --dry-run"
monitoring_URL="https://monitor.io/12345678-1234-1234-1234-1234567890ab"
new="pictures"

$rclone_jobber/rclone_jobber.sh "$source" "$dest" "$move_old_files_to" "$options" "$(basename $0)" "$monitoring_URL" "$new"

If the job specifies new=“last_snapshot”, it behaves like the stock rclone_jobber.sh script.
Backup job script is described in https://github.com/wolfv6/rclone_jobber/blob/master/rclone_jobber_tutorial.org#backup-job-script

Of course, test on a small test directory before deploying real backup jobs.

8

Thanks for responding! I tried this on a small test directory and ran into problems. I edited the rclone_jobber.sh, to replace the new=“latest_backup” variable with new="$7".

The folder I’m trying to use as the latest backup folder is ‘music’, so in the job script, I set the variable new=“music”. The rclone target is “localtest_crypt”, set as $test_remote, which is a subfolder in my home dir called rclone_test.

My script reads:

#!/usr/bin/env sh

### testing using paramters to override the "latest_backup" folder name

#substitute $rclone_jobber and ${remote} with paths on your system

rclone_jobber=$rclone_jobber #path to rclone_jobber directory

source="/home/joti/test_rclone_data"
dest="${test_remote}:"
new="music"
move_old_files_to="dated_directory"
#options="--dry-run"
monitoring_URL=""

$rclone_jobber/rclone_jobber.sh "$source" "$dest" "$move_old_files_to" "$options" "$(basename $0)" "$monitoring_URL"

#display test directories (comment these if calling from job scheduler)
tree -a $source
rclone ls $dest

The output is:

Back up in progress 2018-08-21_21:51:35 _music_test.sh rclone sync /home/joti/test_rclone_data localtest_crypt:/ --backup-dir=localtest_crypt:/archive/2018/2018-08-21_21:51:35 --log-file=/home/joti/github_repos/rclone_jobber/rclone_jobber.log ERROR: _music_test.sh failed. rclone exit_code=7

The localtest_crypt:/ is what I think is wrong, like the rclone_jobber.sh isn’t handling the $new variable in the job script properly, but I don’t know why.

9

The job command needs to pass in the “$new” argument, on the end like this:

$rclone_jobber/rclone_jobber.sh "$source" "$dest" "$move_old_files_to" "$options" "$(basename $0)" "$monitoring_URL" "$new"
10

That did the trick. Thanks again.

12

Hi, I'm a total noob but I'd like to use your backup system. I can't figure out how to use it on Windows.
I downloaded the folder from github, I opened the setup_test_data_directory.sh, but after that it seems that the cmd doesn't know what I'm trying to do.
Any suggestion / explaination on how to run it on Windows (for dummies)?

Btw thanks a lot for sharing your script with the community

13

Hi Jam,

rclone_jobber was tested on Linux, and should also run on Windows 10 wsl https://docs.microsoft.com/en-us/windows/wsl/about
I have no experience with wsl so I won't be able to help you with that.

1 Like
14

Oh I will try that. Thanks a lot for answering a year after your post!