Building Servers for Fun and Prof... OK, Maybe Just for Fun

Jeff, please also note that when you use the Amazon option, you get their ability to host in multiple datacenters, and load-balance across all your purchased instances.

These aren’t small things.

If you roll-your-own setup, you will have to maintain contracts with multiple data centers, and you will have to buy (or implement) your own basic network security and load-balancing devices.

That’s not chump change. It’ll easily add the cost of another server plus open-source routing/loadbalancing/security packages (in your roll-your-own strategy) or the cost of dedicated routers, firewalls, and load-balancers, if you decide to purchase them off the shelf–a thousand dollars to several thousand dollars more in upfront charges.

Plus you’ll have to self-admin servers (with their multiple points of failure) and network security and routing devices, which aren’t necessarily the skillsets that software developers want to spend precious focus on.

Sorry Jeff, I completely failed to read the point that you made at the very end of your post in which you specifically call out that your approach is for folks who “don’t need the redundancy, geographical backup, and flexibility that comes with cloud virtualization.”

In light of that exception, my objection is misplaced.

Our company recently investigate the cost/benefit of AWS. We went with our own colocation facility.

Some salient points:

  1. While AWS is more expensive compared to rolling your own physical servers, the talent required to DIY is expensive. This makes AWS very attractive to startups without someone in-house who can perform the operations-related stuff.

  2. The only way to make your environment cost-competitive with AWS reserved instance costs over a 3yr basis is to virtualize your own datacenter. You can virtualize using Amazon and have them be the beneficiary of your spare CPU/RAM/Disk cycles or you can virtualize yourself and save over the long haul.

  3. For AWS costs to be even marginally competitive over a 3-year basis you must sign up for reserved instances to lower costs. This means paying a large sum of money up-front. If you have 50-100 servers in AWS, good luck going to the bank and telling them you want a loan for something that is a “soft cost”. The bank wants evidence of hard assets that can be recoverable in the event you can’t pay that money back. This makes taking out a loan for reserved instances difficult but makes it much easier to get a loan for your own hardware.

  4. We spend 35K a month currently in AWS. We calculated that even taking into account the fastest possible hardware, an insane amount of RAM and SSD san, hosted in one of the best datacenters in the country, paying for bandwidth, rackspace and salaries, hosting on our own had an ROI of ~14 months if we doubled in size every 6-8 months. AWS becomes very expensive as you host more machines there.

  5. Good luck getting the kind of IOPs necessary for big data from AWS. Their SSD tier does not provide the kind of IO you need for scale and it is very expensive.

  6. AWS is perfect if you need to scale up and down all the time or if you are a startup and have more money than operational technical talent.

  7. We will continue to leverage AWS for some of their ancillary services. AWS is a good tool and there is always room in the toolbox for exotic instruments that can be used at the right time.

  8. The more services AWS hosts, the more complex it becomes. That complexity has a serious operational cost. Have you taken a look at the descriptions for their downtime events? It sounds like a giant Rube Goldberg contraption.

You should include the price of Microsoft Windows Server in your calculation for a fairer comparison.

Hi Jeff, long time fan, first time commenter… I love building servers too and I’ve managed a small group of servers, I personally use Linode, and my currently company uses AWS and some internal servers…

You would agree that in coding you pick the right tool for the job (scientific computing would use a different technology stack than standard ecommerce startup website)…

  1. AWS is elastic (you pay a premium for being to scale up or down - and there’s value to the agility with which you can change or add new services)

  2. AWS RDS is a huge improvement over managing MySQL replication, and they have ELB and lots of other addons that take serious Ops chops to create and maintain

  3. Server operations cost is not the raw hardware:

a. The biggest cost in Ops is people (same as coding), so leveraging Amazon saves on how many people you need to pay to manage your server farm (yes, SysAdmins take holidays and change jobs so cost = N+1 )… you can outsource half way by colocating but the setting up the redundancy, monitoring, auto scaling, etc. becomes a physical pain (you want West Coast and East Coast servers, right).

b. The infrastructure of cooling, UPS, network (bandwidth!), backups, etc. is also a big factor in Operations (does your server room have building security? backup generator?)

My point is that for a stealth mode startup or any internal lab testing buying servers is a no brainer - do it with ESXi or OpenStack and hack away!

BUT for Production you’ll need some Cloud strategy (AWS competitors: RedHat OpenShift, RackSpace Cloud, IBM, ATT Compute, Google AppEngine, etc. means lower prices and improved services)

As you’ve already mentioned if you happen to have hanging around a pile of cash and tech expertise that’s underutilized…

The Server Images which you included in this post are really looking are from the amazing places. Check here the xbrl software vendor details : http://www.taxprintindia.com/xbrl-software-service-mumbai.htm

Omg, loved the picture book cover. Nyce!!!

Jeff,

ditto to Brandon0, and back to the topic of the actual server you built: what RAID controller are you using?

It appears you are using the onboard controller on the motherboard which seems risky since the general consensus, including discussions on ServerFault, are that onboard RAID is ‘RAID in name only.’ i.e. IT, not the drives tend to be the source of catastrophic data loss.

Are you not concerned that you’ll have a lot more to worry about than just the rather poor SSD reliability you’ve written about elsewhere?

Are you getting enough upstream at home for a reasonable price to run a server there?

Apologies in advance - I am that lowest of orders: an end-user and RentaServer renter.

I have been getting screwed by Hosts ever since the Web was invented. (literally)

Numerous SERIOUS marketing efforts have been started over the years and all have ended with crashed Sites.
To explain: I am a serious marketer and learned to drive traffic the hard way in the hard world with pay-in-advance ads.
The Web should have been a paradise for me as a small operator with miniscule costs compared to the “real” world.
Almost EVERY promotion I’ve run - and they are still expensive even out in the Cyberbog - has worked thus resulting in Server crashes from even minor peak traffic.

We aren’t talking Markus Frind figures here, just a few thousand hits.

After all these years, I have never got a straight answer and never had a Server service stay up for one month without downtime.

I don’t need PeerOne, I keep hearing of guys running Servers from home, like Plentyoffish.com did and all these years later, with traffic that would make ME a billionaire on 10% of it, he still runs everything as almost a one-man band on literally, 1/300th of the number of Servers his competition uses.

I had high hopes for the “Cloud” with its lies of distributed loading and and from 1&1 who promoted it heavily, to others, they fall over, so where is this redundancy?
My three current “trials” have all fallen over in the last 3 months - all “cloud-based”…

  1. Run a few Forums.
  2. No high-demand music/flash/video downloads, mainly text.
  3. Have 5000 concurrent users.
  4. Handle spikes of visitors of 10,000 per hour NOT second or minute, per hour.

plentyoffish.com was handling 100 times that with the colossal demands of a dating service system, on a home PC. Running Windows as the final insult! :-}
I don’t even need big data pipes. No videos, no music.

With all the tech expertise I’ve seen on this Board here, someone must be able to tell me the secret.
Or, at least how Markus did it.
Why do others need 600 Servers and 500 staff and he needs a couple of renta-boxes and his girlfriend?

Thank a lot for this blog. I bookmarked it last year knowing that this year I would be building a new server. Now I am ready to build it.

Would you still use the same specs? Or would you move to an Intel E5-2620 (6 core) platform?

My server will be mainly file storage, but I would like to leave the opportunity to expand into Virtualization in the future.

Here’s what we built in 2013

  • Intel Xeon E3-1280 V2 Ivy Bridge 3.6 Ghz / 4.0 Ghz turbo quad-core ($640)
  • SuperMicro X9SCM-F-O mobo ($190)
  • 32 GB DDR3-1600 ($292)
  • SuperMicro SC111LT-330CB 1U rackmount chassis ($200)
  • Two Samsung 830 512GB SSD ($1080)
  • 1U Heatsink ($25)

$2,427

vs. what we’re building in 2016

  • Intel i7-6700k Skylake 4.0 Ghz / 4.2 Ghz turbo quad-core ($370)
  • Supermicro X11SSZ-QF-O mobo ($230)
  • 64 GB DDR4-2133 ($680)
  • Supermicro CSE-111LT-330CB 1U rackmount chassis ($215)
  • Two Samsung 850 Pro 1TB SSD ($886)
  • 1U Heatsink ($20)

$2,401

About the same price, but twice as much memory, twice as much (and probably 50-100% faster) storage, and ~33% faster CPU.

Some load numbers:

  • 2015 Skylake build – 14w (!) at idle, 81w full CPU load
  • 2012 Ivy Bridge build – 31w at idle, 87w full CPU load
1 Like

No concerns about not using ECC memory in the new build?

I like racking servers and the price savings myself, but the reason AWS is killing it in the market is instant provisioning. Getting quotes back and forth with data center people was the most annoying part of the whole thing to me. The hardware folks are great at getting you hardware quickly, but getting it into a rack under a new contract was always a huge PITA and waiting around.

New blog post related to the ECC issue going up today! Keep :eyeglasses: out for it.

1 Like

Over the past few years, to leverage the benefits of instant provisioning which AWS provide, I have implemented VM environments with SALT or Puppet to empower the development teams to provision their own servers. As previous updates mention it is far cheaper to Co-Lo rather than AWS - its up to the imagination of your SysAdmin or DevOps guys to turn wish lists into reality

@codinghorror: Thanks for the write-up but ESPECIALLY thanks for continuing to post newer builds long after the original article. This is very helpful. Thank you.

1 Like

Been following this post for a few years now. And actually looking to build a server. Did you update the server config in the 2020s?

2 Likes

Proudly updated our servers to AMD Epyc, as they are pushing the boundaries of cores (way more!) and ECC for everyone, not just “server branded” stuff. The big wins were

  • 2× amount of memory
  • 8× speed of storage (SATA SSD → M.2 PCI NVMe)
  • 4× number of cores

This is Zen 2 so it isn’t necessarily a ton faster per-thread, but so many more cores, so much more bandwidth, and double the memory.

2 Likes

Curious if you saw any issues with non-ECC memory (following from To ECC or Not To ECC ) in production?

Also, what’s your take on the Ryzen 16C/32T chips – wouldn’t they be faster for running Ruby and JS compared to EPYC?

1 Like

Once you get a lot of servers with a lot of memory, the “insurance policy” of ECC starts to make a whole lot of sense. We’ve definitely had failed memory incidents, both with ECC (!) and without, but the without is a bit more common. At scale ECC is a must have, but I still remain confused as to why the mass market of Dell and Apple and HP are mostly selling machines with regular plain old memory…

“at scale” means, to Dell and HP and Apple, the cost of dealing with RMAs for failed ram is not outweighed by the cost of adding ECC to every system they ship. :thinking:

I guess the logic is

  • servers tend to deal with “important” data
  • there tend to be lots of servers, therefore failure rates as low as 1% start to become painful
  • servers tend to carry heavier loads

so the extra ECC cost is worth it? I dunno. More data here:

image

:point_up_2: think about this… CPU FAILURE IS ABOUT AS LIKELY AS RAM FAILURE! :exploding_head:

Latest data

TL;DR if you have more than 10 machines, I’d get ECC RAM. In the big picture all ram should be ECC, but isn’t… for… reasons…

2 Likes