Incremental and full backup

Hi,

Is there a way I can use rclone to do incremental and full backup and upload backup to AWS S3 or GCP Cloud Storage?

Thanks in advance and I look forward to hearing from you.

Best Regards,

Kaushal

this is what i do, a simplified example of a full backup and forever forward incremental backups.

rclone sync \path\to\source remote:backup --backup-dir=remote:archive\20210121.083621

@asdffdsa Thanks for the reply and much appreciated. I am not sure if I completely understand the below command.

#rclone sync \path\to\source remote:backup --backup-dir=remote:archive\20210121.083621

I have the below source directory which I need to take a full and incremental backup. Once it is backed up (both Full and Incremental) it should be pushed to AWS S3 bucket.

GitLab SCM home directory.
/opt/gitlab/

Full Backup on Sunday late night
Incremental Backup on Monday, Tuesday, Wednesday, Thursday, Friday and Saturday.

Please suggest. Thanks in advance and I look forward to hearing from you.

Best Regards,

Kaushal

hi,

that script will perform a forever forward incremental backup.

i have a script that generates the command, each time with a different time-stamp for the archive sub-folder
the latest set of files is always in remote:backup
each incremental is a sub-folder of remote:archive

each time that sync is run, rclone will compare local files to the files in remote:backup
let's say there is a local file named file.txt that is newer then the same file in remote:backup.
rclone will do the following

  1. move file.txt from remote:backup to remote:archive\20210121.083621
  2. copy the local file.txt to remote:backup

@asdffdsa I have a follow-up question.

that script will perform a forever forward incremental backup? Apologies I have not still understood. Are you referring to the below command?

#rclone sync /opt/gitlab/ remote:backup --backup-dir=remote:archive\20210121.083621

Does remote:backup means AWS S3 and remote:archive is local? I am not sure how this remote:archive\20210121.083621 is generated? I am finding it hard to understand. Apologies for bugging you. I will really appreciate it if you can explain me with some examples. Please suggest. Thanks in advance and I look forward to hearing from you.

Best Regards,

Kaushal

hi,
the remote: is a remote created by rclone, using aws s3.
so remote:archive\20210121.083621 points to aws a3.

i have a python script that generates this command, this case using wasabi, a s3 clone known for hot storage.

C:\data\rclone\scripts\rclone.exe sync "C:\data\u\keepass\database" "wasabi01:en07/kdbx/rclone/backup"  --stats=0 --fast-list --backup-dir=wasabi01:en07/kdbx/rclone/archive/20210201.120322   --log-level=DEBUG --log-file=C:\data\rclone\logs\kdbx_quick_wasabi\20210201.120322\rclone.log

on linux

dt=date +%Y%m%d.%I%M%S
rclone sync /path/to/local/folder remote:backup --backup-dir=remote:archive/$dt


@asdffdsa is the right one for rclone. I too use this method since I really like that my backups are not reliant on another tool (except for if I use crypt). It is a good-enough strategy.

But, also as @asdffdsa mentioned, it is forever forward incremental. What you end up with is your main backup that is a current view and then you have another directory of everything that would have been deleted or overwritten (and keep the logs around so you know which one).

An alternative is something like restic which can use rclone as a backend. Restic works by breaking your files into chunks (blocks) that are decided based on the content and having a database of them. This means that if you make a small change in a big file, the whole file isn't copied again; just the block. The other benefit is that every backup is its own full snapshot and you can delete any snapshot you want and then recover (prune) the space. The (major) downside to this method is the complexity and requirement of using restic to get it all back.

Whether you want that or not depends on you and your own use case. For now, I choose the inefficiencies of rclone for the benefit of simplicity. I just thought I'd put some other options out there.

1 Like

Also, if it helps, below is my backup script:

#!/usr/bin/env bash
#
# Simple rclone backup script. Note that since this is run in cron,
# I've found I need full paths

# cd to the right directory since this is where I have filters
cd /home/<USER>/git_repos/backup_scripts/

NOW=$(date  +"%Y-%m-%dT%H%M%S")

echo "----------------- Starting Backup -----------------"
echo $NOW
echo "---------------------------------------------------"

SRC=/home/<USER>/

DST0=b2:<MYBUCKET>
DST=$DST0/current
BCK=$DST0/backups/$NOW

# Log files
mkdir -p /home/<USER>/logs/backups/
LOGFILE=/home/<USER>/logs/backups/rclone_${NOW}.log

/home/<USER>/bin/rclone \
    --config /home/<USER>/.config/rclone/rclone.conf \
    --fast-list \
    --log-level INFO --log-file $LOGFILE \
    --links `# translate to .rclonelink` \
    `# Exclude the log before the other filters` \
    --filter "- logs/backups/rclone_${NOW}.log" \
    --filter-from filters.txt \
    --exclude-if-present .ignore \
    --track-renames \
    `# --track-renames-strategy modtime` \
    --transfers 25 \
    --delete-excluded \
    "$@" \
    sync  \
    --backup-dir $BCK \
    $SRC $DST

/home/<USER>/bin/rclone \
    --config /home/<USER>/.config/rclone/rclone.conf \
    -v \
    --no-check-dest `# It is a new file. No need to check` \
    copyto \
    $SRC/logs/backups/rclone_${NOW}.log \
    $DST/logs/backups/rclone_${NOW}.log

echo "----------------- Finished Backup -----------------"
echo $NOW
echo "---------------------------------------------------"

well written

very well written :wink:

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.