Sync Script - Check Drive Status First

I’m running rclone nightly to sync my local storage to Google and it’s working beautifully. But as I think about it more, what happens if I lose a drive? If rclone runs and sees a bunch of files missing, it would delete all that info from the remote.

Ideally, I’d catch the bad drive before rclone runs and stop the nightly backup. But I’d like to try and add something to my nightly script to check and make sure all the drives are there before running so I don’t accidentally delete a bunch of files.

Here’s my current backup/sync script:

#!/bin/bash

if pidof -o %PPID -x "rclone-cron.sh"; then
exit 1
fi
rclone sync /video PlexRemote-Encrypted: --min-size 50k -v --stats 0 --min-age 15m --exclude *.csv --log-file=/home/<username>/logs/rclone-sync.log
exit

I use MergerFS instead of RAID, so if a drive does drop out of the “pool”, the files are no longer available. MergerFS handles all of this without issue and I can easily run a sync the other way to get the files back. I just want to make sure I don’t delete a bunch of stuff because I’ve got a drive down.

Thanks!

Hello kyleinprogress,

That’s why we have the --backup-dir option; it makes it relatively simple to recover from the scenario you describe above, and similar situations.

You could process the output of “df /drive” and if the last field (the mountpoint) isn’t where it’s supposed to be, the drive has disappeared and you would abort the backup.

I have no idea how MergerFS works, but if using it means your drive no longer has a mountpoint, you could try to compare “find /drive | wc -l” to the minimum number of files you expect to be there and if it is less than that, likewise abort the backup.

Cheers,
– Durval.

I ran into this problem while debugging RCloneSync
https://github.com/ncw/rclone/wiki/Third-Party-Integrations-with-rclone . My solution for checking the access health of both the local and remote systems is to place 1 or more test files in the same locations on both systems, and check that they are found before kicking off a sync operation. I had one really ugly event before I implemented this check when Dropbox was down once.

There are many different solutions to checking the health of local/remote mounts before syncing. Here are some that I use. I update the mtime on all files when they are first written to disk, set AGE to 65 minutes and run sync every 60 minutes.

# Die silently if there are no new files > 50MB in the local (FROM) directory 
# This prevents syncing when only nfo/jpg files have changed and FORCE is not set
[[ -z $(find "/${FROM}" \( ! -regex '.*/\..*' \) -size +50M -type f -mmin -${AGE}) && -z "${FORCE}" ]] && exit

# Make sure REMOTE is actually mounted
# REMOTE is an ssh-mount so .mounted is just an empty file on the remotes
if [[ ! -f "/${REMOTEA}/.mounted"  || ! -f "/${REMOTEB}/.mounted" ]]; then
    logger "It looks like the remote dirs are not mounted! Exiting."
    exit 1
fi

Rather than syncing one, giant directory, I list sub-dirs and then sync them individually, that way I can skip empty dirs, for example, if the source un-mounts during a sync:

for TODIR in ${TODIRS[@]}
do
    SRC="/${FROM}/${TODIR}"
    [[ -n "${VERBOSE}" ]] && logger "${SRC}"
    [[ ! -d "${SRC}" ]] && continue
    SAVEIFS=$IFS
    IFS=$(echo -en "\n\b")
    for DIR in $(ls "${SRC}")
    do
        # Bail here if there are not files other than *.nfo and FORCE is not set
        [[ -z $(find "${SRC}/${DIR}" \( ! -regex '.*/\..*' \) \( ! -iname '*.nfo' \) -type f) && -z "${FORCE}" ]] && continue
        ###...Your rclone sync command goes here
    done
    IFS=$SAVEIFS
done

I find that pushover/pushbullet/etc are really useful tools for sending me messages when something goes wrong so that I can shut the whole operation down until I have time to deal with it so that, for example, a broken sync does not keep getting triggered by cron. You can integrate them into a BASH script very easily.

There was an issue (can’t find it at the moment) to make rclone refuse to sync if the source directory was empty. That would help here I think.

I think a more useful option would be a --max-delete=NUM option analogous to rsync such that rclone bails if it deletes more than NUM files on the remote during any operation.

There is an issue for that one that links to the issue that you are looking for, which you closed in favor of the --max-delete option :slight_smile:

1 Like