One of the central themes in McConnell's Software Estimation: Demystifying the Black Art is the ominously named Cone of Uncertainty. The cone defines statistically predictible levels of project estimate uncertainty at each stage of the project.
When I was a freshman in Engineering at UCSB, our professor for Engineering 1 told us about the e to pi rule. The idea is that whenever the engineer says how long it will take or how much it will cost, multiply that by something between e and pi to get a more accurate estimate.
I love the reference to the “cloud uncertainty” because i was thinking about the same thing…
I’ve been stuck at 99% of my current project for the last two weeks.
But today, it will be finished, right?
having been involved in various contracting (real, physical building things type) companies, mixed in with this software stuff, what remains true is that real engineers keep track of their experience/history. Such that, at time t=1, they have a set of defined tasks they didn’t have at time t=0, for which they have realized resource requirements. Thus, over time, they accumulate a set of tasks/resource requirements. From this set, call them widgets, they can figure out, with increasing accuracy, what a project (assembled from these known widgets) will cost.
I have not yet, in 30 years, seen a software organization that did this. The universal reason given: “we never do the same thing twice, so collecting performance data is a waste of time”. Thus, all estimates are pulled kicking and screaming from the sphincter of MBA project managers who think Excel macros are programs.
call my cynical.
Personally, I am 99% done with all of the project tasks that were due yesterday. Of course, there are a handful of small outstanding issues left to deal with for each of those tasks, and while I was at it I found a ton of extra tasks nobody ever thought of (do you think I should have my PM add those to the list?)… but hey, I’m on schedule man! 
I have been 100% done with my current project…a dozen times over the last 3 years. The customer is stil waiting for that next “must have” feature before “rolling it out to the entire organization”. The biggest problem is, of course, for every feature they see, they think of 3 other things they want. “Cross this line you die. Ok. Cross this line you die. Ok…”
Buggy, have a look at XP. We use “yesterday’s weather” to estimate - the work we got done in the last iteration tells us how much we can get done in this one.
Think about it this way: if we didn’t firmly believe that we’re 99% done at any given day, we wouldn’t have the courage to go on for the next five years until the project is done!
Though I’m a developer, I can see it, occasionally from the manager’s standpoint. Do you really want your developers saying: “Since this project isn’t exactly like any other we’ve done, we don’t know how long it will take or what resources we will need. Just pay us forever. Can’t promise we’ll ever ship a product.”? There has to some kind of a committment from the developers, ideally based on some experience. Just because human estimates of the future aren’t precise doesn’t mean we shouldn’t plan and commit. Hollywood produces movies and, years later, the “director’s cut” comes out (which, of course, is “done right”).
Hey, I once wrote an After Effects plugin that uses that illusion to create 3D objects from a 2D photo or piece of movie footage. You could set up 2D layers in 3D space and project from another 2D layer onto that one. It was fun to set up a bunch of randomly placed 2D layers in front of some random photo, then move the camera around and look at the distortions from different angles. And it turns out there are real artists who paint stuff like this on sidewalks. Check this out:
http://www.snopes.com/photos/arts/sidewalk.asp
Regarding what barry said, you know damn well that the reason developers don’t want to give an estimate is because an “estimate” suddenly turns into a deadline because managers often treat them as a promise rather than the guess that they are. While it has occassionally bitten me, I luckily have never lost a job over such a situation, but I know others who have. If that happened to me, then you bet I’d be hesitant to ever give an estimate based on past experience again.
If there’s one useful contribution that the Agile folks have made (I think there are actually loads, but stick to this one for now) it’s the Planning Game in XP, which derives from Scrum.
I’ve worked on projects with year-plus timescales where we knew that our carefully-concocted “estimates” were just saying “it’ll take a long time and cost a lot of money”.
I’ve been trying to reduce scope and increase iteration count ever since. Even with my profound optimism it’s hard to get too much wrong when you’re only having to think a week or so ahead.
Buggy Fun Bunny (and everyone): Steve McConnell’s book is basically about how to pick “widgets” for estimating software projects. Also, he discusses both agile and traditional approaches.
It’s a really good book!
I just finished a project phase that spent 6 weeks at 99% done. The relief is indescribable. How about the unimaginably vast swamp of uncertainty?