What are the optimal Settings for big sync job?

Help and Support
8

Here is what I use for backing up to an openstack swift cluster

#!/bin/sh

DEST=remote:backup
DATE=`date -Is`

echo "Backing up, with archive directory $DATE"

# backup home directory
rclone --retries 1 \
       --size-only \
       --fast-list \
       --skip-links \
       --one-file-system \
       --checkers 16 \
       --transfers 16 \
       --backup-dir "${DEST}/${DATE}" \
       --exclude "/mnt/**" \
       --exclude "/.cache/**" \
       --delete-excluded \
       --max-backlog 1000000 \
       --progress \
       sync /home/ncw ${DEST}/current

And here is how I backup with restic

#!/bin/bash

hostname=myhost
email=me@example.com
log=/tmp/restic.log
default_dirs=~

# send output to log and terminal
exec > >(tee -a $log) 2>&1

# A POSIX variable
OPTIND=1         # Reset in case getopts has been used previously in the shell.

prune=0
list=0
backup=1
while getopts "h?pl" opt; do
    case "$opt" in
    h|\?)
        echo "$0 [-h] [-p]"
        exit 1
        ;;
    p)  prune=1
        ;;
    l)  list=1
        backup=0
        ;;
    esac
done

shift $((OPTIND-1))

# read the command line arguments
dir=$@
if [ "$dir" = "" ]; then
    dir="$default_dirs"
fi

function notify() {
    local when=`date -Is`
    echo "${when} $1"
}

# This means use rclone with the directory remote:{hostname}_backup
export RESTIC_REPOSITORY=rclone:remote:${hostname}_backup/
export RESTIC_PASSWORD=XXX

# lower CPU and disk priority
renice 19 $$
ionice -c 3 -p $$

# restic init

if [ "$backup" = "1" ]; then
    notify "restic backup for ${dir} starting for ${hostname}"
    
    # Make the backup
    restic backup \
           --exclude "/mnt/**" \
           --exclude "/.cache/**" \
           --one-file-system \
           ${dir}
    
    restic_exit_status=$?
    
    # mail if failed
    if [ $restic_exit_status -ne 0 ];then
        notify "restic backup failed"
        mail -s "Restic backup failed on ${hostname}" ${email} < ${log}
    fi;
    
    notify "backup complete"
fi

# Tidy the old backups if required

if [ "$prune" = "1" ]; then
    restic forget \
           --prune \
           --keep-last 3 \
           --keep-daily 3 \
           --keep-weekly 3 \
           --keep-monthly 12 \
           --keep-yearly 75

    restic_exit_status=$?

    # mail if failed
    if [ $restic_exit_status -ne 0 ];then
        notify "restic prune failed"
        mail -s "Restic prune failed on ${hostname}" ${email} < ${log}
    fi;
    
    notify "restic prune complete"
fi

# list snapshots

if [ "$list" = "1" ]; then
    restic snapshots
fi

Depending on how you back up

  • if you backup using rclone sync to a plain google drive remote then you will be able to use the files in the webinterface
  • if you backup using rclone sync to a crypted google drive then you will not be able to use the files in the web interface. You will be able to use rclone mount though
  • if you backup using rclone+restic you won't be able to use the web interface or rclone mount. However restic does have its own way of mounting (read-only) its snapshots.
3 Likes