Cheaper than cloud storage - colocating my own storage server

Off Topic
1

Hi there,

as many of you, I have been looking for a new storage solution and I wasn't really happy with the pricing of any of the cloud storage options, so I decided to build my own storage server, which has now been running perfectly for a couple of weeks. As some of you have expressed interest to follow my project, I would like to share my experiences here. There is quite a lot I have to share, so I will divide this into at least two parts.

Let's first get a quick overview of my previous setup, which was working perfectly for the last couple of years: I have a big collection of large files on a google team drive, which I had mounted with rclone on a dedicated server. The server was mainly used to stream the files over the internet. It is also powerful enough to host the occasional CPU-heavy game server (e.g. for satisfactory) and it benefits a lot from Intel's Quicksync, so an Intel CPU with integrated graphics is one of the requirements.

Recently, my team drive has become read-only, so I needed a new storage solution for new files (4 - 6 TB / month). Existing files can stay on the teamdrive for as long as possible, so don't worry, I will still be using rclone :). Everything I store can be found all over the internet and is easily replacable, so I do not really care about data integrity or backups. For my particular use case, I decided that it would be worth it to build my own storage server and colocate at a datacenter. This first part is intended to explain how I arrived at that decision by comparing the cost between different cloud storage options and colocated storage including the hardware that requires. In later parts, I will share more details about the build and any experiences that may help anyone planning a similar project.

Part I: Cost for Cloud vs. Local Storage

Cloud storage

The cheapest trustworthy cloud-storage options that I found were either IDrive e2 or Hetzner storage boxes:
IDrive e2 offers S3 compatible storage for 4$ / TB / month on a flexible plan, or for only 1.67$ / TB / month on a yearly plan.

Hetzner storage boxes vary in price between 3.81€ / TB / month for the smallest box and 2.42€ / TB / month for the largest box (20TB). Boxes can be extended whenever needed, so to save costs I could start with a small box, then extend it, then add another small box, extend it again etc. However, to keep it practical I calculate only with 10 TB and 20TB boxes, as with smaller boxes it would become too cumbersome to manage.

Let's compare the cumulative cost between the options during the first two years. I calculate with 6TB per month, which is the upper end of my requirements, just to give us a little headroom. To compare USD to EUR, I use today's exchange rate of 0.95 EUR/USD.
Unbenannt

The IDrive flex plan is quickly undercut by the Hetzner storage boxes after only 7 months (or 42 TB). However, at the end of the two years, the cheapest (and more managable) option is the IDrive e2 yearly plan. At that point, I would have paid a bit over 4100€ for storage but for the third year alone, I would have to pay that same amount again and so the cost is quickly rising.

So for the long term, would it be cheaper to build my own storage server? I live in a small appartment and don't have the space or a fast enough internet connection to host it at home. Is it feasible to colocate it at a datacenter? Let's try to do the math:

Colocated "Local" Storage

Recurring Colocation Cost

The cheapest colocation option I could find close to my home is at Hetzner in Nuremberg. They offer 14 unit rackspace for 120€ / month + 0.5355€/kWh for electricity and 1.19 € / TB for outgoing traffic over 2TB.
I expect an average power usage of 85 W + 6 W / drive, which would amount to a cost of about 34 + 0.75*NDRIVES € / month for electricity. Outgoing traffic will rarely be more than 2 TB / month, but let's calculate with 5TB / month to be on the safe side, which means ~4 € / month for traffic. Additionally, I would like to have IPv4 connectivity on my server, so I will have to pay an additional 16.18 € for a /29 subnet. Let's also assume 3 trips to the datacenter with my car and 4x 30 min remote hands per year, which would be another 12,50 € + 16,67 € = 29,17 € / month.
Total monthly cost per month for colocation: 207 - 222 €, depending on the number of installed drives.

Server Hardware

I could fit a lot of storage into 14 rack units, but I would need a suitable server case. Many people recommend to get used supermicro or similar server hardware, but those are hard to find and expensive in Germany. Instead, I found a 24 bay hot-swappable 4 unit 19" case on alibaba, which looks exactly like the Inter-Tech 4U-4424 but is much cheaper. The downside of buying on alibaba is that shipping takes a look time (almost two months). The case is compatible with ATX mainboards and power supplies, which allows me to save a lot of money using consumer hardware. I do anyway not care for redundancy for high availability or ECC-RAM. With this case I could build a server that meets (and actually exceeds) all my requirements for less than 1300€, HDDs excluded:

Item Price (incl. tax + shipping)
Case (alibaba version of Inter-Tech 4U 4424) + sliding rails 314.17 €
Mainboard NZXT N5 Z690 Black 109.00 €
2x 32GB G.Skill Aegis DDR4-3000 CL16 RAM Kit 98.56 €
750W NZXT 80+ Gold C Series V2 PA-7G1BB-EU Power Supply 108.99 €
2x 2TB Samsung 970 EVO Plus M.2 NVMe SSD 179.82 €
Intel i5 12600k CPU 249.00 €
Arctic i35 CPU Cooler 22.80 €
2x Dell H200 SAS controller 47.32 €
IBM 46M0997 SAS Expander 33.00 €
4x SSF-8087 to SFF-8643 SAS Cable 0.5m 44.78 €
2x SSF-8087 to SFF-8643 SAS Cable 0.8m 22.99 €
2x SSF-8087 to SFF-8087 SAS Cable 0.25m 21.99 €
Total 1252.42 €

Hard Drives

With a little patience I was able to find 18 TB enterprise HDDs from private sellers with 3+ years warranty. In particular, I bought 4x 18 TB + 1x 20TB drives for 995.49 € (incl. shipping), which should be enough for the first year. I will use 20TB drives as parity drives (more on that in a later part). If these prices stay constant (they are more likely to drop over the years), I expect to pay about 20x 190 € + 4x 240 € = 4760 € for 24 HDDs, giving me 360 TB usable storage (enough for 5 years) and 4 parity drives.

Other One-Time Cost

There are a few more things to buy: First of all, there is a setup fee for the colocation. Moreover, I want a KVM console to be able to remotely access or powercycle the server in case something goes wrong. Luckily, there is a cheap option called PiKVM, which is based on a Raspberry Pi. I have lots of RPis laying around anyway, so I used one of those for "no cost". I will also need a network switch to connect both, the server and the PiKVM, to the internet. A simple unmanged switch would be enough. There are a lot of cheap used rackmount switches on ebay, but they use a lot of power, so I opted for a popular TP-Link 8 port switch and 3D-printed a nice 19" mount for it. I also ordered spares for the SAS-backplanes, because I don't want to wait for them to ship from china in case one fails. Unfortunately, I forgot to order them together with the server case, so I had to pay again for shipping. I guess 4 spares is a bit excessive but whatever.

Item Price (incl. tax + shipping)
set-up fee for colocation and /29 ipv4 subnet 161.53 €
Geekworm KVM-A3 Kit for Raspberry Pi 4 84.89 €
Raspberry Pi 4 0.00 €
Geekworm power supply 20W 5V 4A 12.89 €
TP-Link TL-SG108E Managed Switch 8 Port Gigabit 26.60 €
Rack Power Delivery Unit 17.90 €
4x Spare Backplanes 99.03 €
2x 2m CAT6 Ethernet Cable 7.98 €
M6x16 Cage Nuts 30 Pack 6.99 €
50x #6-32 UNC x 6mm HDD screws 5.14 €
Zivacate 120 velcro straps 6.49 €
Total 429.44 €

Comparison

Now that we have a good estimate for the cost of colocating storage in a datacenter, let's compare it to the cloud options:
Unbenannt
If I intend to keep the data for more than 36 months, it is cheaper to colocate my own storage.
For this comparison, I added the cost of the dedicated server to the cloud storage options (47 € / month). If I colocate my own, I can get rid of that server. Moreover, I assumed that I buy a new drive every 3 months. This would save electricity (and HDD-wear) for unused storage capacity, maximize warranty periods and take maximum advantage of decreasing HDD prices (although the calculation assumes constant prices). In reality, it is more practival to buy a couple of drives once a year.

At the end of the 5 year period, when the server is full, I will have saved more than 4000 € - enough to replace a lot of failed hardware parts, if needed. From then on, ignoring hardware fails and assuming no further growth after 5 years, the monthly cost for the colocation would be 222 € (0.62 €/TB) compared to 571 € (1.59€ / TB) with e2. Even if I assume a 20% HDD failure rate after 5 years and assume that I bought all HDDs at once in the beginning and that HDD cost remains the same - all very pessimistic scenarios - the replacements would only increase the cost to 0.84 €/TB, so I think the risk is managable.

It would have been even cheaper to use Hetzner storage boxes or e2 in the first year and then migrate to local storage when the cost exceeds the recurring colocation cost, but I didn't want to bother dealing with two setups and storage migration.

Also in favor of local storage are faster access times and not having to worry about the upoad/download bandwidth to the cloud. Of course it is possible that cloud storage providers could become much cheaper within the next 5 years, but I doubt they will undercut 1€ / TB. I also entirely ignored the value of my own time and labor, but I personally consider this to be a plus, because it is a fun a hobby of mine that I very much enjoy spending time on.

In part II I will explain the hardware selection and the configuration of the server.

Disclaimer: The prices listed in this post are what I paid at the time and may be out of date.

Tagging @ashlar and @durval, as promised.

9 Likes
2

Thanks a lot for the tag!

I will read avidly.

1 Like
3

are u colo in your own or are u doing with some pp to split the cost?

4

On my own so I have full control over the traffic / power consumption. I am also planning to add another server for a couple of services that are currently running on a quite expensive vps.

5

Great post! Many thanks for tagging me! Looking forward to Part II!

1 Like
6

nice am myself trying to do this as well but due to lack of found i might try to do it with some pp

7

Nice summary, thanks for that!
Been thinking about doing the same but having it hosted @home as I've got a decent upload speed.

I was thinking about just getting an external HDD closure (4-bay probably) and start of with one or two HDDs and keep expanding as required. As I would mostly store non-critical stuff on it, I don't want to spare on HDD as paritiy drive due to costs. But, I saw that Backblaze offers a personal service that allows you to sync your stuff to them for 70 bucks a year.

Problem is, they do not offer a Linux client but I was thinking about spinning up a Windows VM and passthrough the external closure to the VM to use the Windows client instead.

Anyone had a thought on that?

8

BackBlaze offers a service that it calls 'b2' and which works, on Linux, with BackBlaze's own command-line client and with rclone. I myself use it with rclone. Perhaps though I misunderstand what you have in mind; b2 is a cloud service, and seemingly you want to host your data yourself . .

9

I feel like OP overlooked a few options...

  • In the time OP was looking at the prices, you where able to get a auction storage server from Hetzner with 15 * 10TB drives for around 170 Euro per month. That alone was 1.30 Euro/TB.
  • Or a SX64 ...

If we look at todays situation, auction storage servers are expensive but the regular SX65... servers are 1.87 Euro / TB (if one drive is used for Raid 5), or as OP stated, he does not care about the data, that means 1.40TB. If he ran out of space, ... then you move from the SX65 to SX135.

  • Found OP his calculations also very optimistic. I did the same calculations for my own project and that was at home (so no 120 Euro colocation) and 32cent, instead of the 50 cent in power usage. And the bill grew to 0.85+/TB.

I see 0.75 Euro / month per disk in Electricity? How ... A basic 6W * 24h * 30 days is 4.3Kw in power. Doing with my cheaper 32 cent electricity price, makes it 1.38 Euro / month. If we take OPs 53 cent / kwh, that is 2.28 Euro/Month per disk. So unless OP is sleeping a bunch of drives, that 0.75 Euro / month number is way off.

Then there is the average 80W usage ... 12600k, 2x H200, SAS expender... That 80W is not average but the lowball for this configuration because the H200 and SAS expander are going to eat easily 40W+ ...

Then there is the 190 Euro for a 18TB, 240 Euro for a 20TB ... Those are not new drives, and a refurbs so forget about that 5 year failure rate. Each failure is another drive you need to buy, but also remote hand AND rebuild (what can be another drive failure)... That 3 year warranty means nothing from ebay sellers.

In other words, it might just be that OP is break even compared to a just renting a storage server at hetzner (or getting a cheap, at that time auction storage server).

Just taking OP numbers, moving his dedicated 47 Euro server, minus a SX64 96 Euro price at that time, means that he paid 49 Euro for at that time 4x16TB drives. So ... 1.02 Euro / TB (raid 5). Or with a SX65, 124 - 47 = 77 for 4x22TB = 1.16 Euro / TB.

And again, more drives can simply be getting another server or upgrading to a higher capacity one.

Lets be honest, OP, you simply wanted to colocate with your own hardware :wink:

1 Like
10

Thanks a lot for your detailed response. First of all, sorry to I never followed up with part two. I will still do that at some point.

You raise a few valid points, however I still think my solution is cheaper in the long run.

This one I may have overlooked, indeed. I do remember checking server auctions, but I don't think I have seen such an offer. If I recall correctly the SX64 had 4x16TB drives and was about 115€ here in Germany. That's almost 1,80€ / TB.

Those were released 5 days ago... Also, I do not care about the data enough to pay for a backup, but I still use parity drives (4 once all bays are full).

You're right, it should be 2.28 €/Months. I'm not sure what I did there, sorry. Maybe I was thinking about sleeping them back then but I'm not doing that now.

Not sure what to tell you here other than that power usage in the last couple months has been consistently around 75kwh/month, which amounts to an average power draw of 105W. That is with 5 drives installed and includes also a network switch and a PiKVM. Meanwhile, the electricity price has been reduced a bit to 0.476 ct/kwh.

As I said in my initial post, I bought them from private sellers, not from ebay. They are NOT refurbs. I put a lot of effort into finding those and making sure that they indeed are covered by warranty for at least 3 years

Except that the SX64 does not fulfill my hardware requirements on the CPU. I need an intel iGPU. So instead of calculating with 96€ - 47€ (price was higher in Germany btw.) you should calculate with 96 + 47€. Also your calculation wouldn't scale with adding more server, since I only had one 47€ dedicated server.

Of course I did, I would never do such a thing if I had no interest in managing my own hardware. But I do think I'm saving money in the long run.

Btw. the server has been running great since October. I have been paying 170-180€ / month for colocation, IP-subnet, electricity and traffic with 6 drives installed. No failures or downtime so far. No remote hands needed, although I had to make one trip to the datacenter, because I locked myself out of my PiKVM when I updated it, which stopped tailscale from starting up again (known issue that could have been avoided if I head read the PiKVM documentation properly). But we could combine the trip with a visit to the Nürnberger Christkindlesmarkt, which made my girlfriend happy, so that was a plus :smile:

I will try to write part two of this sometime in the next couple of weeks.

2 Likes
11

https://webcache.googleusercontent.com/search?q=cache%3Ahttps%3A%2F%2Fwww.hetzner.com%2Fdedicated-rootserver%2Fsx64%2F&oq=cache%3Ahttps%3A%2F%2Fwww.hetzner.com%2Fdedicated-rootserver%2Fsx64%2F&gs_lcrp=EgZjaHJvbWUqBggAEEUYOzIGCAAQRRg7MgYIARBFGDoyBggCEEUYPNIBCDEwODBqMGo0qAIAsAIB&sourceid=chrome&ie=UTF-8

96.75 Euro (inc 19%) ...

The new SX65 jumped the price up a lot (28 Euro). For what is technically, 4 * 4TB more storage and 2 * 1TB NVME, with 2 Extra Zen2 cores.

I like that you can actually use that low end storage server now for more then just storage thanks to the NVME drives but that was some nice price premium.

Wait when you see how much the older SX63 used to cost! Until Hetzner used the whole Chia HDD mining as a excuse to increase the prices.

SX 63: 82 EUR ( E3-1275 ) with 4x 16TB
SX 64: 96 EUR ( AMD 3600 ) with 4x 16TB
SX 65: 124 EUR ( AMD 3700X ) with 4x 22TB + 2x 1TB NVME

A rather worrisome price evolution in my book. And Hetzner has also been pulling a lot of those SX63 and older auction servers, so they can use them for storageboxs. That is why the action servers are missing. Yea, people love to rent those but hetzner has been slowing the release of those. Notice also the lack of SX64 in the auctions, when SX65 is out.

I admit, been looking more into colocation or another alternative for my project, because of those price increases. Makes it difficult for long term planning.

True ... Still way too darn expensive at 0.476, when residential already dropped to 0.32 for over a year. If only we had actual proper glassfiber here in Germany, instead of this ridiculous 1000/200 for 70 bucks. We have a home in Spain and that has 1000/1000 for 20 Euro!! per month. If we paid 30 Euro, we can get 10.000/10.000. I checked a few years ago, and you where able to get a 1Gbit fiber for like 200 per month (and i mean with a actual SLA, not just a home fiber).

I am rather surprised that your two H200 / SAS expander are not the bulk of your power bill, as those things normally eat 10 a 15W / piece. That 12600K, yea, MB/CPU can idle down to 20W range. I did include your Pi/netwerk etc in my 80W estimate (forgot to write it down). So, close i got lol

Think we all know what your doing with that server (after mentioning the need for a intel iGPU) :wink:

Yea, that is the issue with over protecting a server... The risk of locking yourself out. This has really, REALLY never happened to me before, why are you looking at me like that... really, i never locked down a server to accept SSH from a single IP, and forgot when moving that my IP changed. It really never happened ! :blush:

Will love to read it.

1 Like
12

Well, not sure what you mean by that... :roll_eyes: :grin: and I'm not the OP, but I myself require a iGPU for quickly converting the x264 files I generate with my action cams to x265 for reduced storage.

Strong :+1: to that. @Hoptional sure knows what he's doing and we can all benefit from his experiences and results. @Hoptional, when you post it, please post a link in this thread here so all of us who are following it get alerted to the new one, OK?

1 Like
13

The Chia thing was really crazy. When the initial 'suckerenthusiasts' realized they were being shafted by Bram Cohen &Co. and they would never see any return on the $$$ they 'invested' in hardware (to say nothing of time, aggravation, electricity, etc), a lot of good hardware got dumped and I managed to snatch a nice pair of 2TB Enterprise SSDs still with a lot of life on them and a few 5TB portable HDDs for really good prices.

I wrote "was" above, but I Just had a look at r/chia and it seems that, despite everything that already happened and even many recent, heads-up posts on the (continued) shafting like this one, XCH is still a thing and people are still putting up 'farming' and 'plotting' rigs, discussing optimizations, etc.

I guess the guy who said that there's one born every minute really did know what he was talking about! :grin:

14

I think you guys are thinking too small here... or else I'm thinking too big :wink:
But seriously, I would only look at servers with 8+ HDDs, because (like the OP) I would not trust any data to anything less than raidz2 (so 2 parity disks) and with eg only 4 disks I would be spending 50% on that redundancy.

With that in mind, that's the best I can find right now at https://www.hetzner.com/sb:

image
image843×198 21.4 KB

This works out to (15-2)*10= 130TB usable, and therefore 213/130= EUR ~1.64/TB/mo (in my case, actually 19% cheaper than that, so EUR ~1.38/TB/mo as I'm not in the EU).

Still a little too rich for my blood, and totally un-competitive with the current reliable cloud storage prices, eg iDrive Personal which for 100TB is charging exactly USD 100 right now, so exactly USD 1/TB/mo (and additionaly they have large discounts for first year, and for year payments, etc).

This is with the 18TB/20TB drives you mentioned in the OP, and reserving 2 drives for parity, right? So (and ignoring hardware purchase costs and other initial outlays) we are talking about (6-2)*18= 72TB useable space, so 170/72= EUR 2.36/TB/mo?

This looks incredibly expensive to me (see eg the iDrive costs I mention above, or even the Hetzner auction server costs).

Now I'm sure I'm missing something...

15

does idrive personal support rclone?
"(d) will not access or attempt to access the Services by any means other than those provided on the IDrive® website or by an authorized third party."

1 Like
16

They are rclone official sponsors (or at least were the last time I checked the sponsor list), so if that's not "authorizing", I don't know what is is :grin:

As per the "Personal", my understanding is that it uses the same protocol as their "Enterprise" plans, so there should be no surprises here (AFAIK their not-rclone-supported product is iDrive360, a totally different thing).

17

afiak, only for idrive e2, s3 storage
image

18

iDrive Personal is totally different product than iDrive e2 S3 storage.

iDrive Personal is more like crash plan (which BTW is $90 for unlimited).

iDrive e2 S3 is $4000 per year for 100TB (with first year discounted to $2000)

1 Like
19

Indeed with only 72TB it is quite expensive at the moment, but it scales much better with increasing capacity. I am about to put in the second batch of HDDs, bringing that number down to about 1,30€/TB/month. With all 24 bays utilised, it would be around 0.62€ / TB for 360TB usable storage and 4 parity disks. Of course there is hardware cost on top, but you can see in my initial post how it would play out with that included. And there is still room for two more 4u servers in the rack.

That said, of course it would have been cheaper to start off with one of the cloud solutions and migrate everything over once that exceeds the monthly colocation costs, but I didn't want to bother with that.

20

Of course. Although it's gonna be another couple of weeks, sorry. It has been a busy year but it should be over with an important deadline coming up very soon, so I promise I will find time for it this summer.