--backup-dir pruning code

Sharing some bash code for pruning date stamped --backup-dir directories. You'll have to integrate the below snippet with your regular backup code. It should run immediately after every cron/periodic backup - once its stabilised it should just working in a first in first out arrangement. It will slowly converge to the target # of backups you want to keep, and stay there.

While I think its pretty resilient against over purging, you'll need to decide if you want to carry the risk on your own.

# some header stuff you'll want at the top

# auto_prefix is used to identify cron generated backups vs. manual backups for pruning.  Manual backups are never pruned
# comment out below line for the manual version of the script
# max number of backups to keep (minimum 1).  We will keep backups 1,2,3,...,keep_size.  Older backups are aged out.
# make sure max_keep is something reasonable.  It is possible to zero out the backups over time if max_keep is 0 or negative!!

<do your backup thing with rclone sync --backup-dir, etc >

echo 'Script: aging out old backups'

# credit to various stackoverflow contributors for the general scheme
# approach here is to progressively filter out what shouldn't be purged, and end up with a target list of which a small number may be culled via purge
# this removes a lot of the dependency on time attribute, exactly when the backups were taken and substantially
# reduce the risk of over purging files.

# setting things up
echo 'Script: checking against regex     '$check_regex

# use rclone to pull top level directory, without recursion.  This should give us current + backup dirs + whatever else is in there
# there are some redundant flags c.f. regex to make sure output is good, because we're going to make purge decisions based on it
# if expanding to multiple remotes: MUST use one copy of the code for each remote.  Do not use the purge calcs from one remote on another!

sudo -u kai rclone lsf --dirs-only --dir-slash --exclude "current/" \
    --config='/home/'$owner'/.config/rclone/rclone.conf' \
    $remote1_location | \

# now pipe it to grep with a regex to select:
# 1. entries that which have the $auto_prefix prefix.  current dir will not meet this criterion and is protected.
# 2. entries that conform to the dateTtime format.  current dir will not meet this criterion and is protected.
# 2. entries that end with a slash (i.e. is a directory).
# What is selected *should* be a complete list of all non-current backups and nothing but that.  Manually generated backups, other directories and
# any files at the top dir level should not be selected by the regex.

grep -P $check_regex | \

# now pipe it to a reverse sort which orders the non-current backups in decreasing order by filename (which is the backup date).  i.e. most recent first
# this depends on the dateTtime format being sortable!

sort -r > $tmpfile

# determine the purge count using total # - max_keep.
# wc -l does a newline (\n) count of its input.  the input tmpfile should be clean as long as upstream is clean back to rclone lsf
# wc output should be 0 or greater normally; negative wc output would be ok in terms of not purging
file_count=$(wc -l < $tmpfile)
purge_count=$(( $file_count - $max_keep ))

echo 'Script: file_count is '$file_count', max_keep is '$max_keep', purge_count is '$purge_count

# there are some protective measures here to prevent catastrophic results due to an abnormally high and positive purge_count
# purge_count's inputs may have unintended results, e.g. if max_keep was 0/negative/undefined for some reason
# because of above & other factors purge commands are not looped and only max 2 purges will be executed in a single invocation.
# purge_count = 1 means we're in steady state, so take last (oldest) entry and delete.
# purge_count > 1 means max_keep has been adjusted downwards in a live system.  2 purges are used to gradually bring us back to steady state over time.
# purge count < 1 means we're not yet in steady state.  No purging needed.

# the first purge is if purge_count >= 1; needed in both cases of =1 and >1
if (( $purge_count >= 1 )); then
     echo 'Script: purging oldest backup'
     sudo -u kai rclone --fast-list --config='/home/'$owner'/.config/rclone/rclone.conf' purge $remote1_location$(tail -n 1 $tmpfile)
# the second purge is only if purge_count > 1
if (( $purge_count > 1 )); then
# todo: add in some kind of warning/report that this is happening beyond the echo.  User should pay attention/be aware of this happening
     echo 'Script: purging second oldest backup'
     sudo -u kai rclone --fast-list --config='/home/'$owner'/.config/rclone/rclone.conf'purge $remote1_location$(tail -n 2 $tmpfile | head -n 1)

rm $tmpfile
echo 'Script: done'

Thanks for this. You might want to label the code as being Linux shell code. Indeed it would be good to give the code a shebang (namely,#!/usr/bin/env bash or #!/usr/bin/env sh).