Well I think the comments show more similarities than difference between mid to large scale software development and other more “physical” kinds of development/construction.
I suspect that software is more burdened with unrealistic expectations. The customer compares custom development with the effort required to select, acquire, and deploy shrink-wrapped products marketed to 5,000 to 5,000,000 users and has no visceral feel for the effort that’s amortized over all those sales.
Of course software project is like any project and can be managed! For me this type of thinking is typically a developer’s way to glorify and mystify softare development.
True, there are some never-been-done-experimentary-really-hard-to-estimate ventures. However, an overwhelming majority of projects do not fall into this category. Most SW projects are about organizing GUI elements, storing and fetching data from database, integrating some 3rd party components, implementing business logic and flow. In most SW projects there is nothing new or groundbreaking in the code. It is the business function that might be revolutional.
These projects are not impossible to estimate or design. Difficult, sure. But if there are good requirements and good design, implementation is mostly “just run-of-the-mill work”. Work that is often inspired and fun, creative and challenging. And sometimes really tedious 
But still, saying that most software is unproven and experimental, is to me a far stretch.
It seems that many developers have some kind of natural aversion to proper design and implementation guidelines. This typically leads to chaos, odd experiments and delays, but the reason why most IT projects fail is not that software development is too difficult.
Ok, off my high horse Thank you for your thought-provoking blog.
Coming from both an aero-space and software background, I can tell you that both the risks and challenges in the development cycles are extremely similar.
No airplane is identical, and even within a given line (eg: F-16) there are many blocks which differ substantially in how the components integrate with each other. Their software analogy would be version upgrades.
Next up from that are variants (A 747-400 is substantially different than a 747-400ER), which is similar to the differences between say Visual Studio Standard and Enterprise editions. Each variant of a 747, while based on the same historical airframe and air-tunnel research, starts branching off and adding different features based on their unique target markets.
Hardly anything in aero-space engineering is repeating the same thing over and over. From the airframe designers to the line engineers, there’s always a new challenge they have to incorporate with their existing knowledge.
And funny enough, the high-risk, experimental airplanes are the staunchest adherents to the waterfall model.
Conversely, a move towards away from the pure waterfall model can be found in Multi-disciplinary Design Optimization (MDO), which appears in more mainstream engineering efforts. Here, they are attempting to iterate more frequently in order to achieve better use of materials and structures. The waterfall models do attempt some iterative loops, but they’re low in number and generally pre-determined.
So, why has the aero-space industry generally use the waterfall model and gantt is planning overall projects? Because the correct use of these require only two pieces of information:
That is all that an accurate (if not entirely precise) estimate requires. Any more information and it’s likely a biasing factor because of wishful thinking or political reasoning. Techniques such as MDO makes it harder to estimate a project with any reasonable accuracy because the dependencies added from the feedback loops cannot be enumerated. How can anyone before hand, say how many iterations will it take, before we get it right?
For the aero-space industry, the cost savings of MDO promises to make it cheaper in the long run for production to offset this uncertainty in development cost.
So, can software development learn anything from the aero-space industry? Quite a lot I must say. The same risk-analysis, and structured development methods can easily be applied to software development.
7 Where it makes sense from a long-term cost, use MDO. (*2)
Footnotes:
*1 - Ironically, despite the fact that most programmers should have studied network theory, the software industry is the only one where I frequently see people try to do a 60%-40% rule or some obscene optimistic fudging for schedule+risk estimates. Scheduling is a longest path problem – any fudging is called wishful thinking. If a value is too long for your liking, plan to answer the risks early enough so you can re-estimate, not arbitrarily change the value.
*2 - My personal opinion on #7 is that frameworks and language designs are probably the only application of this rule. And as every good developer should know, unless you’re in the frameworks business, you should not be building frameworks.
To echo Dave Mackersie comment that “Software never leaves the design phase”, have a look at this article from 1992 called, “What is Software Design” http://www.bleading-edge.com/Publications/C++Journal/Cpjour2.htm
“I concluded that we are not software engineers because we do not realize what a software design really is. I am even more convinced of that today.”
Even though the article was written 14 years ago, I believe it still holds true today. I wonder how many of us that write code realizes that, "A program listing is a document that represents a software design. Compilers and linkers actually build software designs.” Seems obvious, but…
As Brad Cox (inventor of Objective-C) wrote 16 years ago in his article, Planning the Software Industrial Revolution, “I only wish that I were as confident that the changes will come quickly or that we, the current software development community, will be the ones who make it happen. Or will we stay busy at our terminals, filing away at software like gunsmiths at iron bars, and leave it to our consumers to find a solution that leaves us sitting there?” http://www.virtualschool.edu/cox/pub/PSIR/
Software Industrialization is one of my favorite topics when I am not filing away at iron bars
Software Engineering? “A study akin to numerology and astrology, but lacking the precision of the former and the success of the latter.” I wish it weren’t true, but coming from an electronics engineering background, it seems like we have a long way to go for software development to be called “engineering.”
a href="http://developer.valvesoftware.com/wiki/Valve_Time"http://developer.valvesoftware.com/wiki/Valve_Time/a
Right…
P.S. Be sure to check out my recent post on my blog about this subject - blog at ggglog.B_L_O_G_S_P_O_T.com
By the way - take out the underscores (_) in the URL of my blog.
This problem is why we have object/component oriented programming. Although a piece of software may not exist, most of the components needed to build it do. I oversee custom software projects for a large team and I’m usually fairly close on all of my project deadlines. We do a lot in audio/video processing and it definately helps when you stay around the same area. I imagine if we set out to build something radically different, my estimates wouldn’t be as close.
I’ve noticed something interesting. People seem split into two camps.
Here’s my two American cents: I work for a large company and I have done lots of work on well established, large software packages. When adding new features or module enhancements to established packages, estimation is much easier and waterfall/BDUF works surprisingly well. As a software publisher, we’re responsible to a large customer base, so having a good design helps. There aren’t many surprises, so the waterfall isn’t a hindrance. Projects, when well managed can come in close to time and budget.
Since the company is so large, I’ve also done a lot of work in more cutting-edge areas on newly established products. Waterfall and BDUF aren’t so good in that arena. A more agile approach is necessary since the plan tends to change much more often. Nobody (the company, managers, or developers) has a large stake in the new product, so rapid and large change decisions can (and do) get made often. That’s okay… a professional developer should be able to be agile enough to handle it (in my opinion). Estimation of completion at the start of that project is next to impossible. What to do? Work with management and provide ongoing status updates often. Make sure that they are aware of how all change decisions are affecting the time line and where the project stands and give re-estimates often.
My opinion is free and probably worth what you paid for it.
g
As a software engineer who has worked closely electrical and mechanical engineers for many years, I think the key difference is this:
Software never leaves the design phase.
Now, I know that as software engineers, we often make a distinction between design and coding. But unlike mechanical engineers, our work never needs to go to a manufacturing floor to get built. For the mechanical engineer, the transition from design to manufacturing is an important milestone, and represents a significant commitment of monetary resources. In fact, each step in the waterfall reperesents a major decision point to commit more financial resources.
In software, we can implement the work in relatively tiny incremental iterations, with each iteration representing a relatively tiny financial commitment of a programmer’s time. Of course, a lot of those tiny iterations can quickly add up to a lot of money. But my point is, there is the potential for a lot finer grain control over the project, for someone who is able to stay on top of it. I don’t think any other type of engineering discipline can do the same, but if they could I’m sure they would.
I think, finally, Dave Mackersie hits on the key point: Software is an iterative process. That is all.
And whoever Bill is, your little speech about what defines art and what is an artist is frickin’ hilarious. Your romantic vision of yourself is so funny I threw up a little in mouth.
Every time I hear somebody comparing software building with say physical house building and criticise “Well, people have been doing houses for ages, why can’t coders do it well and on time?”, I remind myself the house my parents bought took three times more to deliver, and probably costed much more than original planned, though we didn’t have access to those figures.
The problem lies in that software is a new invention every time and in building it you invariable “improve” and change what you had done in the past. Nick Bradbury (a href="http://nick.typepad.com/blog/2006/06/start_coding_li.html)"http://nick.typepad.com/blog/2006/06/start_coding_li.html)/a describes it like this, “… after I’m comfortable that the memetracker is working as intended, I’ll then throw the code away and start a more formal design approach. Cowboy coding is a great way to find problems, … the goal here isn’t to complete a feature quickly (although that’s often a side effect), but instead to discover the gotchas up-front so you can design the feature correctly.”
You often don’t see the gotcha’s even in a well defined project. In the case of a kitchen remodel or any remodeling of an older home, you don’t know what is inside the walls/floor/ceiling. You don’t know if Jimmy did the wiring himself and daisy chained it or cross wired something. It’s the gotcha’s that happen all too often in software that change things. The 3rd party tools and widgets don’t work as advertised and you have to build your own. The customer doesn’t want to spend for the widget and you have to change the design to meet what you have, etc. Sure the construction process might be the same, but getting the individual parts to work together isn’t.
Engineering (like build planes) and software follow same risk patters.
It is risky to build a copy of a working plane ?, really not. It is risky to make a copy of a working running software ?, really not.
It is risky (very) to build a new designed plane. As it is risky build new designed software (a lot less than a plane).
What is different : Thousands of new software are make every year, a handful new planes are designed every year, ergo, new software problems arise often, not new plane problems.
I work for a large company and I have done lots of work on well established, large software packages. When adding new features or module enhancements to established packages, estimation is much easier and waterfall/BDUF works surprisingly well.
Agile is useless for large projects. Agile pundits even say so themselves. Waterfall is fantastic when you have a large team. There’s really no other way, actually.
It’s just that when the team is small ( ~10), the overhead of waterfall can be greater than the actual project itself.
All this talk of construction, projects, and engineering is confusing everybody. Software development is much more like making a movie than any of these.
All the effort is up front and then once released (we even use the same word) you start seeing some income.
Once done, you can sell the work again and again for almost no cost.
Just as many adventure films have been made, so have many accounts payable systems, each with its uniqueness.
If a movie’s a hit the imitators arrive from all directions, just like in software development.
You can patent software about as well as you can patent a movie. Same applies (in the other way) about copyright.
Both have the same attitude about deadlines.
Both industries are legendary for cost overruns and massive flops.
Both industries are legendary for cost overruns and massive flops.
Hey gal’s/guy’s, can you tell me what is the difference between CPU processor ENGINEERING and Software ENGINEERING, or maybe Chinese WATER HYDROELECTRIC POWER ENGINEERING, this are BIG projects… they all take long time, there are some really silly things - like taking one year to write several hundred lines of code like in previous Microsoft rant 
BTW if you are building a web site DON’T take a dozen of developers - man that will eat your soul and your pocket out… like someone before said don’t try to emulate a Michelangelo if you ain’t one – go to school!!! – then work for several years as an apprentice and if you don’t cut it in 3 to 5 years with hugely popular and PROFITABLE project then man/woman hide under the rock and die as a poor programming schmuck and don’t droll and be jealous of people who make a killing in software business or any other…
I read a good description of one key difference between software and engineering projects once: when you’re designing a new engine you don’t usually have to worry that the pistons will leave the cylinder block, destroy another component and pop back in again. And when physical components break, they tend to break in predictable ways.
Don’t be a pussy. Software is no more challenging than machine design. I’d sit in the A/C and write code all day if I could. Programming is the easiest part of the job.
Ya know, I completely agree with you. Software development isn’t easy, but then again neither is say, designing/laying out/manufacturing a processor. There just aren’t many outspoken hardware blogs whining about how their profession has it rough :). No one I know that is really good at it got to where they are by professing REAL hardware engineers started when they were 9 and live/breathe/eat hardware engineering. That’s just dumb and I’m sick of it from the software community.
Software development (and I say development, we are still very much in the dark ages) still has a long, long way to go.
You buy a plane, you buy software. Engineering methods are still needed even if software engineering is project-like. You cannot blame a project failing for being a unique project, if you do not use proper engineering methods. A piece of software is still a piece of software - so software engineering methods are needed to keep that piece intact and proper. You also can recycle smaller parts of program code from other projects. Then there are design patterns in software engineering, but design patterns already existed in for example building architectures.
There really are lots of successful unique scientific projects in the world. Science always builds on earlier science, and new software builds on earlier software. We rarely need to start from scratch, at least we have high level programming languages. The language providers are the basic level inventors. Then there are application level inventors who make basic components. And then we have some trivial stuff like packaging, which feels like assembly lines.
Because we build unique programs, we get money on the unique level, so we can afford good methods. Though there are lots of basic programming, too. Not every program require tons of inventions a’la interstellar travel, but tons of normal features. And even if a software project was more like inventing some mind exploding algorithms, the more important it would be to use scientific methods together with the engineering methods.
If there are lots of things in the project like software, hardware, operating systems, people, paper, and business processes, then these all should be considered - with engineering and with science.