I feel like colocation and bandwidth charges will be the lions share of the total cost and are completely neglected in your comparison. It will be easily several thousand a month for colocating several servers plus a good connection.
âBut to be totally honest with you, mostly I build servers because itâs fun.â
I would hope so, because thereâs certainly no financial reason to do so. Economy of scale is a Thing That Exists. So is TCO. Youâre ignoring both in this article.
The main problem with your estimate is that youâre using 100% utilized instances, but billing them as On-Demand. Granted, thatâs how Amazon sets up the sample, but thatâs not what youâd do in practice.
Youâd use reserved instances, not on-demand, which are a good deal cheaper even taking into account the reservation fee.
When I switch the 6 servers to reserved instances, the bill comes down to $825/mo.
That brings the calculation to $30k for 6 servers for 3 years, almost half less than your estimateâŚ
Taking this a little further: if your colo bill is ~$500/mo, then add $3k/server * 4⌠That brings the 3 year cost of building your own to⌠$30k. Exactly what Amazon charges, strangely enough.
And thatâs assuming none of your hardware breaks. And not factoring in the cost of build time, tweaking, shipping to colo, installing OSâs and configuring, and everything else you have to do when you build your own.
Iâd call that pretty much break even on the cloud hosting, assuming you consider the two builds equivalent. If not, what would you consider equivalent?
Hardly âprohibitively expensiveâ.
Does anyone else find it comical that the links that Jeff provides to the server hardware all go to Amazon and he is quoting AWS? Amazon wins no matter what one does since they sell everything!
Iâve tried colocation, but the server you own becomes out of date faster than it depreciates. Also, Iâve found the colo providers to be very stingy with bandwidth. AWS can get very pricey, unless youâre using them as a PaaS provider. On the other hand, some of the dedicated server offerings are extremely attractive. For example, BurstNet is offering unmetered 1 Gigabit or dedicated/guaranteed 100 Mbit bandwidth for only $179/month, including this high-end dedicated server configuration:
- Quad Core Xeon x3220 (2.4Ghz)
- 8GB RAM
- (2) 500GB SATA w/Hardware RAID
- 5 IP4s included, plus IP6
- Pretty much any Linux distribution, or Windows Server for a little extra per month
Check out http://bit.ly/burstsale for this and other options on sale.
Good point about reserved instances. If I change the (Amazon provided!) default Web Application settings from on-demand to three year reservations, I get 825.17 per month with a 3,600 up front payment:
http://www.codinghorror.com/.a/6a0120a85dcdae970b017c3299638d970b-pi
That breaks down to $925.17 per month. So instead of buying one faster than almost anything you can even buy for any price at EC2 server every two months, you can âonlyâ buy one every three months.
Also note that weâre paying a little bit exorbitantly for the 512 GB SSDs and the top-of-the-line Xeon quad-core CPUs. If I bump that down a bit, I could easily build these servers for around $2k each with almost no meaningful loss in overall performance. (but some loss in storage capacity, yes.)
If I do that, then weâre back to buying one faster than almost anything you can even buy for any price at EC2 every two months again. 
Also for the record our colocation costs, including bandwidth and power are more like $125 per server per month, nowhere remotely near $500 per server! Even that is a little bit expensive, I could probably have done better if I shopped around a bit. Actually now that Iâm comparing notes here:
http://blog.pinboard.in/2012/06/going_colo/
âThey offered me 8U at 2 amps for $200/month with a 100 Mbps unmetered data planâ
âFor $206, I got 10U, 4 amps, and 100 Mbps connectivity in a San Jose data center about 14 miles from Fremontâ
I need to re-negotiate that deal! Our servers are all 1U so we only need 5U, and perhaps 4 amps.
Jeff,
Any change in heart in the aftermath of Sandy? Trello is down and Iâve seen Stack Exchange and Fogcreek rolled over to their back up data centers. Granted AWS and other cloud providers have their outages but it seems like they are shorter than when your primary DC goes down.
While reading all the comments I could not get my thoughts together, I now think I see what is going on.
Some people are spending a lot of time learning how to get good value out of xxx hosting company choosing the correct type of instances to rent etc., other people would rather spend their time understanding how servers work and build their own servers. (Then you have the time to find a good colocation setup if you build your own servers.)
I now thank that EC2 offers a few main benefits.
a) You can scale down without having to take a big loss by un-capitalizing your hardware investment.
b) A small monthly bill you are not commit too, is a lot easier to get approved then buying lots of machines.
c) If it all goes wrong you can blame someone else! I think this is the best reason to use EC2âŚ
d) If you only need very low powered servers for a short time, then EC2 cannot be beaten.
e) You can believe you can stale up next week if needed without having costs this week.
Forgive me for butting in the middle of your EC2 cost discussion, but Iâm building something very similar to this and I was curious what you are using for your RAID configuration. Did you go with onboard RAID, software, or hardware, and why? Iâve been reading lots of people saying to avoid onboard RAID, but I just recently had a hardware RAID fail likely due to the card (3ware 9650SE-2LP). Iâm also doing a Windows build, so there doesnât appear to be a solid software RAID option like Linux has.
Jeff, please also note that when you use the Amazon option, you get their ability to host in multiple datacenters, and load-balance across all your purchased instances.
These arenât small things.
If you roll-your-own setup, you will have to maintain contracts with multiple data centers, and you will have to buy (or implement) your own basic network security and load-balancing devices.
Thatâs not chump change. Itâll easily add the cost of another server plus open-source routing/loadbalancing/security packages (in your roll-your-own strategy) or the cost of dedicated routers, firewalls, and load-balancers, if you decide to purchase them off the shelfâa thousand dollars to several thousand dollars more in upfront charges.
Plus youâll have to self-admin servers (with their multiple points of failure) and network security and routing devices, which arenât necessarily the skillsets that software developers want to spend precious focus on.
Sorry Jeff, I completely failed to read the point that you made at the very end of your post in which you specifically call out that your approach is for folks who âdonât need the redundancy, geographical backup, and flexibility that comes with cloud virtualization.â
In light of that exception, my objection is misplaced.
Our company recently investigate the cost/benefit of AWS. We went with our own colocation facility.
Some salient points:
-
While AWS is more expensive compared to rolling your own physical servers, the talent required to DIY is expensive. This makes AWS very attractive to startups without someone in-house who can perform the operations-related stuff.
-
The only way to make your environment cost-competitive with AWS reserved instance costs over a 3yr basis is to virtualize your own datacenter. You can virtualize using Amazon and have them be the beneficiary of your spare CPU/RAM/Disk cycles or you can virtualize yourself and save over the long haul.
-
For AWS costs to be even marginally competitive over a 3-year basis you must sign up for reserved instances to lower costs. This means paying a large sum of money up-front. If you have 50-100 servers in AWS, good luck going to the bank and telling them you want a loan for something that is a âsoft costâ. The bank wants evidence of hard assets that can be recoverable in the event you canât pay that money back. This makes taking out a loan for reserved instances difficult but makes it much easier to get a loan for your own hardware.
-
We spend 35K a month currently in AWS. We calculated that even taking into account the fastest possible hardware, an insane amount of RAM and SSD san, hosted in one of the best datacenters in the country, paying for bandwidth, rackspace and salaries, hosting on our own had an ROI of ~14 months if we doubled in size every 6-8 months. AWS becomes very expensive as you host more machines there.
-
Good luck getting the kind of IOPs necessary for big data from AWS. Their SSD tier does not provide the kind of IO you need for scale and it is very expensive.
-
AWS is perfect if you need to scale up and down all the time or if you are a startup and have more money than operational technical talent.
-
We will continue to leverage AWS for some of their ancillary services. AWS is a good tool and there is always room in the toolbox for exotic instruments that can be used at the right time.
-
The more services AWS hosts, the more complex it becomes. That complexity has a serious operational cost. Have you taken a look at the descriptions for their downtime events? It sounds like a giant Rube Goldberg contraption.
You should include the price of Microsoft Windows Server in your calculation for a fairer comparison.
Hi Jeff, long time fan, first time commenter⌠I love building servers too and Iâve managed a small group of servers, I personally use Linode, and my currently company uses AWS and some internal serversâŚ
You would agree that in coding you pick the right tool for the job (scientific computing would use a different technology stack than standard ecommerce startup website)âŚ
-
AWS is elastic (you pay a premium for being to scale up or down - and thereâs value to the agility with which you can change or add new services)
-
AWS RDS is a huge improvement over managing MySQL replication, and they have ELB and lots of other addons that take serious Ops chops to create and maintain
-
Server operations cost is not the raw hardware:
a. The biggest cost in Ops is people (same as coding), so leveraging Amazon saves on how many people you need to pay to manage your server farm (yes, SysAdmins take holidays and change jobs so cost = N+1 )⌠you can outsource half way by colocating but the setting up the redundancy, monitoring, auto scaling, etc. becomes a physical pain (you want West Coast and East Coast servers, right).
b. The infrastructure of cooling, UPS, network (bandwidth!), backups, etc. is also a big factor in Operations (does your server room have building security? backup generator?)
My point is that for a stealth mode startup or any internal lab testing buying servers is a no brainer - do it with ESXi or OpenStack and hack away!
BUT for Production youâll need some Cloud strategy (AWS competitors: RedHat OpenShift, RackSpace Cloud, IBM, ATT Compute, Google AppEngine, etc. means lower prices and improved services)
As youâve already mentioned if you happen to have hanging around a pile of cash and tech expertise thatâs underutilizedâŚ
The Server Images which you included in this post are really looking are from the amazing places. Check here the xbrl software vendor details : http://www.taxprintindia.com/xbrl-software-service-mumbai.htm
Omg, loved the picture book cover. Nyce!!!
Jeff,
ditto to Brandon0, and back to the topic of the actual server you built: what RAID controller are you using?
It appears you are using the onboard controller on the motherboard which seems risky since the general consensus, including discussions on ServerFault, are that onboard RAID is âRAID in name only.â i.e. IT, not the drives tend to be the source of catastrophic data loss.
Are you not concerned that youâll have a lot more to worry about than just the rather poor SSD reliability youâve written about elsewhere?
Are you getting enough upstream at home for a reasonable price to run a server there?