With rclone you can easily make a full backup of your server, thanks a lot to everyone involved. But daily, full backups to the same target mean that if you accidentially delete a file, and first discover this a few days later, the file can't be recovered. Instead, you can make a full, daily backup to a new target each night, but this is slow and takes up a lot of space. Here is a small script I use for rotating, incremental backups.
The script makes a remote copy of selected directories to Amazon S3, in the directory recent . When a file is changed in recent , the previous version is moved to one of a rotating set of remote directories. If you have a set of 7 rotating directories, and you run the script daily, you will be able to go a week back in history and find your missing file there. If you just want the most recent version of your files or directories, you can find them in recent .
The first time the script is run, it will most likely take a few hours. After this, it will be quite fast. 10 minutes for my server.
I will be happy if others can use the script and thankful if someone can help to improve it.
#!/bin/bash
SERVER=$(hostname -s)
REMOTE="amazonS3:${SERVER}"
MYDIR="/home/e-smith/files/users/holck/dev/rclone/"
FILTER="filter.txt"
BACKUPLIST="bak.list"
VERBOSE="" # Add more v's for more verbosity: -v = info, -vv = debug
LOG="/tmp/rclone_log.txt" # Temporary log file
cd $MYDIR
# bak.list is a file with names of extra backup-versions, starting with the oldest
OLDEST=$(head -1 $BACKUPLIST | sed 's/ .*//') # the oldest, extra backup-version
COUNT=$(cat $BACKUPLIST | wc -l) # rotate the file, deleting the first entry and adding a new entry at the end
((COUNT=$COUNT-1))
tail -$COUNT $BACKUPLIST > $BACKUPLIST.tmp
echo $OLDEST $(date) >> $BACKUPLIST.tmp
# Purge oldest version
date
echo "Purging oldest version ($OLDEST)..."
nice rclone purge ${REMOTE}/${OLDEST} ${VERBOSE} &>> ${LOG}
echo "...done"
# Perform actual backup
date
echo "Performing backup to ${REMOTE}..."
nice rclone sync / ${REMOTE}/recent --backup-dir ${REMOTE}/${OLDEST} --filter-from ${FILTER} ${VERBOSE} --skip-links --update --use-server-modtime &>> ${LOG}
echo "...done"
date
mv -f $BACKUPLIST.tmp $BACKUPLIST
cat ${LOG} >> log.txt
Here is a sample filter:
+ /etc/group/**
+ /etc/gshadow/**
+ /etc/passwd/**
+ /etc/samba/secrets.tdb/**
+ /etc/samba/smbpasswd/**
+ /etc/shadow/**
+ /etc/ssh/**
+ /etc/sudoers/**
+ /home/e-smith/**
+ /opt/search/**
+ /root/**
+ /usr/lib/mailman/aliases/**
+ /usr/share/wordpress/wp-content/**
+ /var/lib/mailman/**
+ /var/log/**
- *
And a sample bak.list file
old5 Thu Aug 6 01:30:01 CEST 2020
old4 Fri Aug 7 01:30:01 CEST 2020
old3 Sat Aug 8 01:30:01 CEST 2020
old2 Sun Aug 9 01:30:01 CEST 2020
old1 Mon Aug 10 01:30:02 CEST 2020
old7 Tue Aug 11 01:30:01 CEST 2020
old6 Wed Aug 12 01:30:01 CEST 2020