The Big Ball of Mud and Other Architectural Disasters

Mistakes are inevitable on any software project. But mistakes, if handled appropriately, are OK. Mistakes can be intercepted, adjusted, and ultimately addressed. The root of deep, fatal software project problems is not knowing when you're making a mistake. These types of mistakes tend to fester into massive, systemic project failure. That's why I'm fond of citing McConnell's list of classic mistakes; I find it helpful to review every so often as a sort of triage self-check. I ask myself-- am I making any of these mistakes without even realizing it?


This is a companion discussion topic for the original blog entry at: http://www.codinghorror.com/blog/2007/11/the-big-ball-of-mud-and-other-architectural-disasters.html

This essay, along with a bunch of others, is evidently a part of the book series “Pattern Languages of Program Design”

Link to Amazon seach for the five (?) book series:
http://tinyurl.com/2axe47

Guilty of 2., 3., and 4. and hoping for a 7.

Shantytown + Urban Sprawl describes the IT dept at my last employer. PC techs were brought in with no real experience or training and expected to support thousands of PCs with hundreds of apps in the mix. IT dept on the upper end of double digits (in terms of employees) but anything that wasn’t specifically requested by the CEO was on a shoestring budget. Anything not specifically owned by an in house developer was either unsupported or thrown on the PC techs.

What about “Wrap it up” as #8?

Essentially, a solid, well-designed piece of code forms the core of a business and its methodology. Over time, business needs change, but the development knowledge has mostly left. Any current efforts are merely wrapping around the old code because the business can’t/won’t reconstruct. To them, that’s too risky. Meanwhile, the newcomers to the industry have reconstructed three times and are now kicking butt…

So, it’s a bit like DC. Early on, the city was well-planned and manageable. After a while of growth without the discipline to stick to the original vision, and the entire city sprawls 45 miles in all directions in one giant, over-crowded metropolitan area. It was only a matter of time before the “diamond-shaped utopia” that L’Enfant designed and left to grow becomes wrapped and hidden under years of garbage.

Why are these supposted to be negative? I consider many of these to be sensible approaches to realistic software development. I can’t believe that you don’t use them yourself.

  1. Shantytowns.

One part of shantytowns is that they are built using inexpensive tools. What’s wrong with that? I regularly use open source products in my development. I shouldn’t use gcc because it’s free? I shouldn’t use emacs because it’s free? cvs and svn are evil because they can be used at no cost? Are you on drugs? A cheap tool doesn’t make the end product bad. A paint brush is a cheap tool, and da Vinci has made some inspiring works with it.

  1. Throwaway code.

I have no problems with throwaway code as so much of the open source world starts off as throwaway code and this seems to be a successfully approach in this arena. If you didn’t have people quickly throwing their ideas together to have a look at them then innovation will die or at least crawl very slowly. Of course I judge open source projects differently from the software in a 747. Different approaches for different projects.

  1. Piecemeal Growth

Or as I call it: modular design. You take a large system, like a city, and break it up into seperate and smaller areas with specialised functionality, linking them all up with roads. This is basic stuff - if a programmer doesn’t know/use this then he shouldn’t be programming.

  1. Keep it working (Daily build)

Seriously, you’re suggesting that maintance of software is bad? That daily builds are a bad thing? I’m sorry, I have to ask if you’ve ever worked at all in the software business? Nothing is ever perfect. As time goes on bugs are exposed and need to be addressed. This is why we write documentation and comment our code so that we can maintain it latter.

  1. Shearing layers.

I’m not sure what you’re trying to say here. Are you saying that software changes? If so, then why’s that bad? Software has version numbers for this very reason. That software can evolve and grow is a potential force for good.

  1. Sweeping it under the rug.

Again, do you have any real programming experience? How often do we reuse libraries that have bugs in them and we create a facade or wrapper to go around this library. This wrapper tries to minimise the damage these buggy libraries do. It’s a practical reality. Hiding problem libraries is not a sign of bad design, it’s good practice.

7 Reconstruction

Here’s a big WTF. So you’re saying that refactoring is bad? Are you seriously saying that your code is so damn perfect the first time that you don’t have to go back and improve on it? Sometimes the original code sort of almost works, sometime you just have to completely trash it and start again. Refusing to acknowledge a mistake and letting it live on in the form of crap code is unbelievably stupid.

For the record, the Chernobyl sarcophagus is not “sweeping it under the rug,” it’s an attempt to contain the dangerous shit within than would otherwise escape and is so dangerous that it can’t be removed. I’m not sure if there’s a programming metaphor there, really (it’s hard to find a metaphor for code that is really analogous with radioactive sludge).

(The real Chernobyl metaphor would be management related, in my opinion—the real problem the Soviets had, aside from an inherently dangerous reactor design, was in letting vastly under-qualified people monkey with something quite powerful and potentially dangerous.)

Andrew writes, “Again, do you have any real programming experience?”

I was asking myself the same thing about you Andrew. Not only have you completely missed the point of each and every one of the pathologies, you missed the point that Jeff didn’t actually come up with these ideas himself.

“Not only have you completely missed the point of each and every one of the pathologies”

Fair enough. Stupidity strikes again. Let’s start with the first one. In it he quotes: “Shantytowns are usually built from common, inexpensive materials and simple tools.” My question is why is the use of cheap tools a problem with shantytowns?

“you missed the point that Jeff didn’t actually come up with these ideas himself.”

Err. He repeats these ideas in his blog. His blog is an outlet for what he has to say even if what he writes here is a quote. Isn’t it fair to consider that he agrees with the quoted text? He doesn’t go: look, here’s some opinions that I disagree with. Without the disclaimer one can only assume agreement.

Andrew, I think that you actually have missed the point with a lot of your commentary, especially the shantytown analogy. Using cheap tools does not make something a shantytown. The important part was this:

“There is little specialization. Each housing unit is constructed and maintained primarily by its inhabitants, and each inhabitant must be a jack of all the necessary trades. There is little concern for infrastructure, since infrastructure requires coordination and capital, and specialized resources, equipment, and skills. There is little overall planning or regulation of growth”

How did you miss that?

“I was asking myself the same thing about you Andrew.”

I’m bitter, angry and I hate you all. Isn’t that proof that I’ve worked in the software development industry for a long time?

Of course, having worked in software long enough to be completely disillusioned doesn’t mean that I’m a good software developer.

I’m betting Jeff is infinitely better developer than I. At least he still has that innocent joy about the industry. That virginity will pop soon.

“How did you miss that?”

So you agree that the use of cheap tools is not a mistake? So why include that with the description of a worthy mistake? The best writing is to the point. You say what you have to and stop. Extra wordage only runs the risk of damaging your main point.

In this case a real mistake is mixed in with something that wasn’t a mistake. Jeff should have cut and paste less text. Less words doesn’t make a point less powerful.

Didn’t Jeff have a post on this very issue just recently? Something about making the web a place full of rich information and not noise? (Maybe I’m thinking of a different blog - I’m old and my memory is going.)

I’m bitter, angry and I hate you all. Isn’t that proof that I’ve worked in the software development industry for a long time?

Works for me. Welcome to the team.

Hey Now Jeff,

Interesting post, always good to keep in mind revisit every one in a while.
Coding Horror fan,
Catto

Andrew, I also think you are missing the point, but I am not trying to attack you – merely to explain how I read the example.

Shantytown means (all of this is my opinion alone) that you take many separate parts, and each one is responsible for doing too many things. Like if I made an app composed of 5 different technologies. Each one had need to send an email. Instead of having one piece of code that handled all email sending capabilities, each part of my shantytown would all have to know how to create an email, communicate with a mail server, handle errors, etc.

I’m really not attacking you, I am trying to explain what I took from the Shantytown example.

Oh – and I’ve been involved in plenty of shantytowns in my day.

I’d seriously like to find out from other software developers if anyone has been involved in a project that you were very proud of the code. I’ve been working at various companies for 10 years or so, and at every one of them the project I’ve been dumped into was 1, 3, 4, or 6. I have done some throwaway code things, too, but I’m more concerned with big software right now.

My first job was probably the most organized place I’ve worked for. They had a big (50k) PowerBuilder 5 project that was used in small co-operatives around the country helping farmers plan how to plant their fields (and therefore spend more and smarter money with the co-op). PowerBuilder MADE you develop OO code, and still there were large parts of the program that were done very poorly – several places where code was repeated, in a slightly different manner because someone didn’t have time to learn how to use another’s classes correctly.

Another job had a massive health care application that all the developers on were dying to rewrite because they felt the code was a mess of spaghetti.

Another one I took over from a developer who had maybe 10k lines of code written, which were sufficient to do the task he was given. I then was asked to add more and more and more functionality, and made some bad choices so eventually I was left with 70k lines of code that made my stomach ache when a bug was found. I hated going back in and trying to piece together what to fix, how, what it would affect, etc.

At my current job, I’ve got 70k lines of code that I’m not proud of, but it is pretty easy to maintain. Some parts are a mess because I didn’t know the .Net framework at all, and ended up doing things REALLY badly since I was under so much pressure to deliver functionality so quickly. I am working on refactoring all of this. However, I doubt that I’ll ever be “proud” of the code. Who knows.

So, has anyone worked on a project that was planned (or refactored) so well that maintenance is a joy (relatively speaking), new features can be added logically without screwing up the main plan, etc?

Hey, it’s a very clean Shantytown. Here is one that looks more like some systems I’ve worked on: http://www.flickr.com/photos/krazzy/2059259925/

I’m pretty sure Andrew nailed it with the Shantytown / Open Source comparison:

“There is little specialization. Each housing unit is constructed and maintained primarily by its inhabitants, and each inhabitant must be a jack of all the necessary trades.”

Hrm…sounds like open source development to me…

“There is little concern for infrastructure, since infrastructure requires coordination and capital, and specialized resources, equipment, and skills.”

interesting…just like open source…

“There is little overall planning or regulation of growth”

…aaaand again.

The reason “big ball of mud” approaches are so prevalent is because they work. I would go so far as to say it’s the only thing that works for projects that need to evolve (read: 99% of software). Anyone who thinks their current project is not a big ball of mud, is either delusional, or simply hasn’t shipped version 2 yet.

copy + paste = blog!

All software wants to become a ball of mud! It is its natural state. All emergent systems look like mud in the end, anyway.

I’ve spent 5 years maintaining a massive, 10-year old VB6 product for about 10 to 20 thousand customers. No automated tests, no documentation, comments you shouldn’t dare trust, no consistency. None of the original developers are around anymore, and at least 40 developers have stomped around in it from around the globe. New releases are about 8 months apart.

To fix bugs and add features, I must forget everything I know about traditional software design:

  • Code Duplication is king. It’s the only way to prevent breaking the unknown.
  • Don’t comment or write documentation. Doing so will only tempt novice engineers to “fix” what is currently there.
  • Only design for the immediate feature request at hand. Don’t fool yourself into predicting future requirements and saving the world.

A favorite line I’ve heard since I started working on the project is, “Don’t worry, the code won’t be around for much longer”. Hah! It will exist for as long as the product continues to bring maintenance revenues.

Under this system, it is expensive to add any new features that require changing the mud. So we don’t change the mud. We build around it.

Embrace the beast and learn to work around it. It may be a shanty town, but it’s still home to someone.