His table with 3 billion rows, actually flows into a much larger observation.
I’ve seen this in every company I’ve worked for. Subprime mortgages, healthcare, insurance, finance, etc. You start an app and it works great. With each passing month you have more data in your database. Eventually you have 1 million records and you start looking at indexing, then it’s 5 million and you start looking at upgrading hardware. Then you have 50 million and you start fine tuning queries and such.
The longer you stay in business, the more it costs you to handle the transactions. Now presumably during this same time period, you’re also automating other things and building economies of scale. This whole time the database is growing, we’re talking about how can we archive off stuff, but nobody really wants to take responsibility for that answer.
It’s like there is a natural progression where the older a company gets the more inefficient it becomes, and this allows for a competitive advantage to a young startup.
That is, unless you do something about this problem.
And it’s not just data, it’s a lot of stuff that you collect over time as a company. Could be boxes of paper. Could be a fleet of airplanes. Just interesting.
I’ve been a developer, and in the computer industry, for over 20 years. The one thing that I’ve learned well is to keep things simple. Less “pieces” leads to more supportable and lower cost solutions. More “control” leads to more efficient development, implementation, and changes. The old KISS statement is where it is at. Limit the libraries you rely on. Cut the architecture to only what is necessary. Scale and performance come from understanding your application and its data intimately and matching your architecture to your solution. And, I hope to become as successful as the aforementioned owner has been with my site at http://freedatingscene.com .
Even though this blog is pretty old now, I have to say in terms of plentyoffish’s desgin, not a lot has really changed over that time. In fact for such a large reputable dating site such as it, I’m astonished as to the astonishing lack of design features. I’ve developed a new site based upon useability to give the user the best possible chance of finding a partner online, which can be found MingleMonkee: http://www.minglemonkee.com
The growth at Plenty of Fish has continued unabated it seems. In recent post Markus proclaims surpassing 6million logins in 1 DAY! http://plentyoffish.wordpress.com/2011/05/16/ Includes a graph to illustrate the last 4 years. I marvel not only at how he’s been able to scale but also the consistent growth of traffic. Wealthy Affiliate