Rclone as destination for Borgbackup

Help and Support
9

Hello everyone, and specifically @ncw,

I strongly recommend against using restic right now, at least to anyone trying to back up a large number of bytes and a large number of files (I have ~60M files here, occupying ~32TB of space on the source file systems being backed up), because:

  • restic backup uses a ton of memory when you have a large number of files (to the point of 64GB -- yes, you read that correctly: 64 gigabytes -- of RAM not being enough).
  • restic prune is mandatory: if you let your restic repo run too large (eg, a month of daily backups, with just ~1M files and ~50GB changing on the source), your remote repo will get corrupted (ask me how I know... :expressionless: ).
  • restic prune currently takes a ton of time and uses a ton of memory: even with on an 8-vCPU + 128GB RAM Google Compute node I created specifically for the prune, after 2 full days running, it aborted with OOM... :expressionless:
  • restic development is pretty much stuck right now: I'm not complaining (on the contrary, I'm grateful to fd0 and the other restic developers for all the time they spent on the project) but it's simply not moving forward; for example, a patch to make restic prune minimally workable in large-repo situations is stuck for over 2 months now, except for people reporting repo corruption issues. Another patch to likewise make restic restore workable for large repos is stuck for almost as long, despite lots of reports (including mine) that it's working perfectly and it only needs to be approved for merging.

So, despite having invested literally months trying to get restic working on my setup, I'm sadly being forced to give up on it :expressionless:, and I'm moving to borg (BTW, that's how I found this topic).

I think you meant rclone sync, right? Otherwise, lots of useless gunk would remain on the rclone remote after each borg prune.

Anyway, that's exactly how I'm proceeding here: I finished setting up borg last night and since then I'm running a borg create to back that 62M files / 32T Bytes source to a local Borg repo, and simultaneously I'm running rclone sync on a loop copying it to Google Drive, something like this:

while ! rclone -v --checkers=12 --transfers=2 \
               --multi-thread-cutoff=1k --multi-thread-streams=8 \
               --low-level-retries=1000  --rc \
               sync LOCAL_BORG_REPO GDRIVE:BORG_REPO; 
do 
    sleep 5; echo === `date`; 
done

So, it just runs one rclone sync after the other, until one of them finishes successfully; not exactly what I wanted, which would be to stop the loop after an rclone sync runs without copying anything (because the local repo was not modified, ie, the parallel borg create command has finished.

How could one go about checking that, without having to parse rclone's output? Perhaps a specific exit code (enabled by a separate option like '--special-exit-if-nothing-copied' so as not to break compatibility with current scripts)?

Thanks,
-- Durval.