Just a Little Bit of Software History Repeating

I lived in the Denver area at the time Denver International Airport's completely computer automated baggage system was unveiled in 1994. The troubled development of this system was big local news.


This is a companion discussion topic for the original blog entry at: http://www.codinghorror.com/blog/2008/03/just-a-little-bit-of-software-history-repeating.html

I don’t really understand the point of this post?

Maybe Sean’s lack of understanding comes from his being a project manager at Terminal 5?

I suspect the point is: learn from others mistakes, life isn’t long enough to make them all yourself.

I could have put money on one of my favourite bloggers commenting on this. I read two key points :

  1. Software engineering is a lot harder than people think - especially non techies. Unfortunately non techies are sometimes the ones writing the checks, and without techies on hand to give authority on what you can and can’t do, you’re going to have a disaster on your hand. Say what you like about Bill Gates but he had enough technical chops to avoid major disasters like this (at least until the mid nineties anyway).

  2. Steve McConnell’s classic mistakes are truly classic. I’m not sure how many came into play with Heathrow T5, but “shortchanged quality assurance” and “insufficient risk management” leap out. You see them playing out again and again. I’d bet money on another IT disaster cropping up in ten years time from this with the same classic mistakes.

I don’t really understand the point of this post?

I know a few have mocked my propensity to bold text before, perhaps rightfully so. But in this case, if you’re wondering what the point is-- just look for the bold text.

I understand that hubris is a key ingredient in many failures. What I was expecting to see, I suppose, was something relating this to software engineering and design.

I thought the exact same thing when I heard about their ambitious “new” project. Oh well, we’ve come a long way in the past 10 years, right?

Open the luggage bay doors, Hal…

…and I’m so worried about the baggage retrieval system they’ve got at Heath Row…

I’m just glad to see the return of gratuitous self-linking. You’ve written gazillions of little posts over the years – the dozens of links, internal or not, are what make this site so easy to get absorbed in.

Also, anyone who doesn’t get the point of this article IS the point of this article.

Is baggage handling systems an eternal curse?

Denver - 1994
Hong Kong - 1998 (http://www.legco.gov.hk/yr98-99/english/sc/sc01/papers/chapter5.pdf)
Heathrow - 2008

Actually it sounds like they put too much faith in the technology working, and didn’t put enough into the human aspects of interacting with the technology or just in moving between jobs. They should have been fully-staffed dry-running the new terminal for at least a month - dispatching fake aircraft and moving real bags back and forth, with the real staff doing the jobs - and pushing back the ‘release date’ every time the tests failed. That would have caught:

  • Inability to park
  • Inability to get into the building
  • Knowing where to pick up baggage from for a given flight

What they actually did was give existing handlers from Terminal 1 five days’ training on the new system, but not really using it in practice, then told them to turn up at T5 instead of T1 last Tuesday morning. Result, everyone tries to park in the same space (and circumnavigating Heathrow to park somewhere else is really hard as the roads jam up completely with even the slightest incident on the M4) and no-one can get to work on time.

There’s a reason that West End and Broadway shows have many days (weeks, months) of rehearsal before opening, including a couple of dress rehearsals and technical rehearsals so that the costume, props, sound and lighting people all know where they have to be, what they have to have ready at what time (including which performers need to be waiting in the wings), and what their cues are. It’s why the military do live-fire exercises. It’s so that when the time comes to put on the performance, or go into battle, it’s just like the last time you did it, only this time you have an audience.

Actually getting a new system into the hands of the users can be a real problem, which is another reason why the ‘big bang’ approach to deployment is such a bad idea.

As I understand it, the baggage handling system worked rather more efficently than the baggage handlers could cope with (mainly for the reasons in Mike’s post above) resulting in baggage backing up to such an extent that the whole system became unworkable. I believe they are currently trying to clear a backlog 15,000 pieces of baggage!

Just FYI, the text doesn’t appear bold to me under Firefox (v 2.0.0.13).

The problem is the real world.

Computer systems that have to interact with a physical entities in the physical world have complexity caused by the unpredictability of the stuff that they deal with. And baggage systems have to cope with real physical stuff.

Imagine a document management system with a workflow. The paths that the document can take through the work flow is a simple permutations and combinations problem based on easily predicatble inputs and outcomes. So it is easy to code.

A suitcase can slide off the conveyor, can get trapped and cause a logjam. One suitcase can land on top of another and go round the system in tandem. A suitcase can fall off the conveyor and be thrown on to the wrong conveyor by a handler. Suitcases can do all sorts of stuff that is difficult to imagine in the office. This makes it difficult to code for all eventualities.

In general writing software for systems that involve handling physical stuff is inherently much much harder than writting software for more abstract stuff.

This by no way means absolves the terminal 5 fiasco which was predicatable and could have been avoided by a sensible testing and dress rehersal (good analagoy mike) prior to go live.

Interesting post, this is huge news over here (London) at the moment, in the paper every day. In fact its such big news, you’d think a commercial airliner or two had crashed into a building killing upwards of 3,000 people.

No, just delays because of baggage handling issues.

I do, however, fully expect all problems to be sorted out quickly and quietly, and we’ll be back to front page articles of Kate Moss’s latest escapades with a magical nostril clogging white powder.

“controlled by 140 computers”

Er… that’ll be the problem, right there - next time, it might be an idea to have it controlled by ONE computer!

I would have built a scaled down version of the terminal using trained mice as baggage handlers and marshmallows as bags then run it into the ground for at least 3 months…

DIA was a failure before it even started. Blame that on politics and a typical government project management failure. DIA is a huge, beautiful and fully-functional airport, but it’s still just an airport - not the modern, state-of-the-art, automated, “all weather” facility we were promised. Blame that solely on the well-greased politicians, not on a “software project failure”. Cut the programmer community some slack, Jeff. :slight_smile:

People working in the new terminal could not park
People working in the new terminal had never been there before and got lost
People working in the new terminal had never seen the actual systems and so failed to work them properly
Some people had had little or no training …(because it was not thought necessary)

They seem to have forgotten that People are part of the system and you need to test the system with the people who are actually going to use it