Cloud Backup Newb - options and costs and when API calls and early delete costs become an issue for backing up Restic Repos

Help and Support
1

What is the problem you are having with rclone?
Problem is trying to decide on how rclone interacts with dfferent types of backups and therefore how this will effect the cost of using certain provider AWS, Wasabi and Backblaze b2

What is your rclone version (output from rclone version)

latest

Which cloud storage system are you using? (eg Google Drive)

that's what I want to figure out

Hi all, I'm new to online backups and have traditionally used a very basic system of onedrive to backup important files on my pc's. I now run a business and use Restic to make weekly backs ups from each of my computers to a removable drive. I now want to complete my backup process by having a cloud back up of my restic repos. Currently I have 3 Restic Repos that I update weekly, each repo is around 150GB and has around 20,000 files in each repo.

My plan is to use Rclone once per week to sync the updated repos up to a cloud provider (AWS, Wasabi or Backblaze b2). My concern is api call and early deletion costs. If using AWS then based on Storage costs it only makes sense to use Glacier class which has a 90 day early deletion period and api call costs. Wasabi also has a 90 day early deletion period but no api call costs, backblaze has no early deletion period but api call costs. Ideally I would prefer to use AWS directly, cutting the middle man as it were.

My understanding of Rclone sync is that it will make 240,000 API calls to check the data (20,000 files per repo x 3 caried out 4 times per month) then depending on how many files need updating a put for each updated file and a delete call for the original file that was updated. Because Restic repos are encrypted my guess is that this will be a really high number of changes to the files so my guess is in total could be around 1million api calls per month? I know I can use --fast-list to limit the calls but here would still be a large number of changes needed? So that is one consideration for cost and points to Wasabi being the favourite option.

However, next is the early deletion costs, syncing 60,000 files, 4 times per month, will mean a lot of early deletion costs. If the repos are changing each week, my understanding is the rclone sync would be deleting and replacing a lot of the destination repo files every time I run a sync, so I would very high have monthly early deletion charges. If the data is being overwritten every month, then my understanding is that the the early deletion costs of using AWS Glacier and Wasabi just make them absolute non-starers? It appears to me that this type of backup strategy of weekly updating incremental backups simply cannot be run on Glacier and Wasabi due to early deletion limitations or am I missing something?

The cost estimate for s3 standard 500GB with 600k api calls is around $15 per month. Backblaze B2 is $5 for 1TB plus around $5 in call costs (1million per month).

These costs seem pretty high to me for disater back up data that I will hopefully never have to access so I'm really thinking that I must have this wrong somehow? It cost me about half that for 1TB of onedrive space that automatically syncs for me with no worry about api call costs or deletion fees, so I'm now thinig I really must have this all wrong. All I need to do is zip my repos to container and replace them on onedrive (any alternative such as sync.com) and it is all taken care off at half the price, so what am i missing. I really want to be able to get a good, cost effective cloud backup solution but think I am have gone wrong somewhere in my understanding and calculations of how it all works.

I would really appreciate if someone could help explain where i am going wrong with my understanding and pricing estimates.

Many thanks

Edit: Just to expand on this a little further, even if I just decided to buy 2TB on aws Glacier with 4 calls per month, just to upload complete copies of the repos and then delete after the deletion period it would still costs more than the ondrive/sync.com example. AWS would cost around $10 per month while sync.com is $8 for 2TB and all syncing taken care off. This can't be right?

2

hello and welcome to the forum,

each use case is different and hard to generalize.
for one month, i would try multiple options and then know the exact costs going forwards.

if you have lots of files that will be deleted then go with onedrive but it has performance issues with syncing lots of small files.

in my use-case, for backups, files do not change once uploaded and stay uploaded for long periods of time.
so i use a combination:

  • wasabi, for recent backups.
  • aws s3 deep glacier for long term backups.

as a side note: i upload veeam backups flies to wasabi, so the retention period is 30 days, not 90 days.
and that applies to all files.

there would be a lot more than 4 api calls per month.

3

Thanks for your input asdffdsa, much appreciated.

Do you (or anyone else) have any experince of using rclone to regularly sync Restic Repos and whether or not my understanding of the scenario is in fact correct? Both Restic and Rclone are mentioned a lot as a joint solution for backing up to the cloud so I would like to get an understanding of how user make it work, and make it cost effective.

With the OneDrive type of scenario I wouldn;t be uploading lots of small files, I was just going to use 7zip to stick them in an uncompressed zip folder as a container and upload he conatiner as 1 large file per repo.

Backblaze do a personal backup drive type offering with unlimited data for under $6 a month (yr upfront). Just boggles my mind that this can be provided at such a much cheaper cost that direct online storage?

4

that is what i do, 7zip a bunch of smaller files and upload it.

but once uploaded, the files will not be deleted for a period of time, 30/90/365+ days.
so retention periods are not an issue for me.

i do not backblaze but most providers, the more you upload, the more they slow you down.
and most users never really upload huge amounts of data.

as for restic, never used it, i use 7zip of small files and veeam backups.

5

Do you mind some unsolicited advice? If not, feel free to ignore. And note that this may be controversial!

TL/DR: Make all of your backups 100% independent of each other. Simplicity, even at the cost of efficiency, can be a good thing for backups.

Don't sync backups including syncing restic repos.

If you want to use restic as a backup tool, it can talk directly to B2. Or you can do it through rclone to make the connection but that is not the same as using restic to backup to, say, an external and then rclone to back up that to the cloud.

Why? Because you lose the independence of the backup. You are protected from losing the disk or something but if your restic backup gets corrupted (which also isn't unheard of) you have not synced the corrupted copy.

It is much better to have your 3-2-1 strategy be totally independent of each other. Use Borg to backup locally (since it can't use cloud) and then restic to backup to the cloud.

Here is where I have gotten a lot of flak but I actually think rclone directly with --backup-dir is a nice backup. It is way less efficient than Borg, Restic, Duplicacy, Kopia, etc. But with efficiency comes complexity and dependance. Ignoring encryption, with rclone, you can use any other tool to restore files. No need to read a database. And broad corruption isn't an issue since there is no single point of failure. (even better if using with B2 is to make an API that can't delete; only hide). Restoring to a point-in-time is much harder and there is little-to-no deduplication but it is just so simple and that's also what you want with backups!

6

Thanks Jwink3101, I appreciate the input.

I had considered the --backup-dir function to move, rather than delete objects within Wasabi to get round the early deletion fees but my concern was the amount of changes that would be made as a result of the sync and therefore the extra data that I would be carrying for 90 days. If I have to replace 50% of synced files every back up then it will easily take me over my storage limit pretty quickly and if I delete it I get charge for it anyway. What is your experience of how much data gets "removed/moved" to the backup dir per sync and the average size that you are syncing? Sync to B2 wouldn't have the deletion restriction in any case so there is no cost implecation of actualy delting with B2, however the API calls could mount up, doubling my monthly bill and that is where My concern with B2 comes in. What is your experience of how many API calls are made per backup and avg size of backup?

I agree with your point regarding independent backups but that is not an option as the desktops are not online for me, so it needs to be a backup of a copy of the restic repos.

For my issue, it all comes down to how Rclone functions when carrying out syncs of Restic repos. I really need to understand what is the average API calls per 100gb synced and also how much of the destination backup is generally ovewriten on average when syncing an encrypted restic repo .e.g. do I have to factor in 1 api call per file int he repo or would it be much less, do I need to factor in 100% of the synced repo being overwrtten (deleted) ever time I back up due to the repo already been encrypted or is it much less and why? If I know these two issues I can work out whether or not it is cost effective to use Rclone.

I'm not really sure what the real world benefits would be to using Rclone direct to a cloud provider now given there is not going to be a significant cost saving either way. I assume there will be time savings from syncing parts rather than uploading full backups (again depends on amoutn that requires re-syncing) each time and possibly that the upload speeds may be better direct to the cloud providers rather than using a cloud sync drive type of service. I've spend the last 3 days really trying to get a full understanding of the whole rclone process and understanding cloud providers, have even set up a AWS account so i'll be a little dissapointed if it is all for nothing. Going back to my original reason was that I thought rclone direct to cloud providers such as AWS was access to storage in an efficient and super cheap manner but I guess this may not be the case due to api call and early deletion costs.

The all important info i really need is 1) Average number of api calls per sync of an encrypted restic repo per 100GB 2) Averge number of files replaced per sync of an encryted restic repo, when repo size is around 100GB

7

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.