As people have pointed out, giving an estimate with a range of several orders of magnitude is generally a non-starter in most organizations.
But I think that the point of the exercise is that if you are requested to honestly and to the best of your ability produce an estimate with a 90% confidence range and the result covers several orders of magnitude; then, for a well functioning organization, this should be a crystal clear signal that significantly more research needs to be done before the project is undertaken.
Of course, most of us work in disfunctional organizations where inconvenient truths are not welcomed. And I would hope that other chapters of McConnell’s new book will cover what to do when estimation reality collides with management fantasy. (Does it Jeff?)
Unless you are an expert on all of the subject matters above, you couldn’t hope to give an accurate estimate on them. The point was to show that if you were forced to, would you be able to account for the inherent uncertainty.
If I was asked to give an estimate on how many lines of code a transaction system would have, without any more information, my estimate would have to be a broad range. The results of this experiment, however, tell me that most people’s wouldn’t be broad enough.
Bill nailed this one: if people simply give a fixed point or too narrow band because it is the corporate culture, that culture is avoiding the problem of the underlying uncertainty.
If the developers are giving wide estimates, it means the problem is not well understood. If better estimates are wanted, more research will be required and a more thorough understanding of the problem needs to be developed.
The idea that most organizations don’t collect enough data to estimate correctly assumes that those organization that do collect and use data are successful at estimation. While I don’t have the statistics, I do have empirical evidence. I can see the results from Microsoft, Oracle, Sun, Developer Express and many others, many of whom maintain statistics about code estimation and yet continuously miss estimates. Microsoft is the poster child for missed delivery dates, even when slashing features.
Do I think the task is impossible? No. But I think that many of the development methodologies make the cost of achieving good estimates prohibitively expensive. I much more prefer an agile/xp approach which is only concerned with much smaller estimates, and therefore can be much more accurate since there are fewer unknowns.
These kind of analytical estimation problems are often referred to as Fermi Questions (named after Enrico Fermi who used to ask such questions of his students).
Do I think the task is impossible? No. But I think that many of the development methodologies make the cost of achieving good estimates prohibitively expensive. I much more prefer an agile/xp approach which is only concerned with much smaller estimates, and therefore can be much more accurate since there are fewer unknowns.
I work for a defense contractor that typically works on 1 million lines of code or more projects. The agile/xp approach just doesn’t scale up for projects that size but you have to come up with an estimate to win the contract. In this case being good at estimating is a competitive advantage. I have even heard that when the country of Italy puts out a work request, they take all of the companies estimates and average them. The company closest to average gets the contract.
I don’t think becoming good at estimating is a huge cost, it is just hard to stay disciplined and actually collect the metrics needed for the next project while you are working on the current one. Everyone says that having previous project metrics is important for making accurate estimates, but do you think Microsoft is keeping track of the number of overtime hours their employees are working to get XP out the door? I would guess not.
In most real businesses, the idea of giving estimate ranges multiples of orders of magnitude wide to management are simply a fantasy.
That’s right. So the next logical question is, given the high level of uncertainty at the beginning of any software project, how do you eliminate or reduce that uncertainty?
It’s all about gathering data-- both as you go, and before you start, through research. Ideally, you’d have some historical project data from your own organization.
Just looking at the number of overbudget projects that continue to occur tells me that you have more chance of creating an accurate estimate using a dart board as you do using any of the estimation techniques used over the last 40+ years of software development.
You’re saying that software estimation is impossible. I don’t think that’s true. It’s a very hard problem, but it’s not unsolvable. I think the main problem is most organizations don’t gather enough data from their past and existing projects (bugs, bug fix rate, function points, etcetera), so they’re starting from a blank estimation slate every time they launch a new project.
The length of the pacific coastline is not well defined. Coast lines have a fractal dimension greater than one, and thus the measured length increases without limit as the measuring stick used grows smaller. Any value above 50000 miles is therefore correct for some stick-length.
The one other commenter who made the same point made the mistake of coupling the observation with the ridiculous and wrong statement that the volume of the great lakes is also not well defined.
Whatever reference produced that precise value for the Pacific coast length is either: a) teasing, or b) another instance of that special mixture of self confidence and ignorance that plagues so much of our world.
What is interesting about the study results are that we have a marked tendency to underestimate (if our estimation is 30% accurate is debatable). How many times have you estimated that a project will last less than what it actually did? My interpretation of this 90% vs. 30% anomaly, is that our estimations (excuses for generalization) are about a third of the real value, when we haven’t researched enough on the problem at hand, as it turns out to happen in the design of this quizz. That doesn’t mean, of course, that with a decent research we cannot approach more to the real value, and possibly that’s the core subject of the following chapters in the book.
Most of the comments here are interesting to read but a bit off the mark. I wonder how many of the people making the comments actually read the book. The “estimation” exercise was part of a bigger discussion of “estimation, targets and commitments”. Estimation, according to the author, is used to help the planners gauge the likelihood of having a successful project. Of course, estimates will be done by qualified individuals. Of course, the narrowness or breadth of the ranges matters. Of course, some people will know some of the answers. None of that is the point! The point is that a huge percentage of people who took the test were asked to get 9 out of 10 questions right yet the vast majority could only achieve 2-4 correct answers. Those of you who blame “management” need to look in the mirror and READ the test instructions.
The quiz is stupid. Most of the questions involve estimating things that are incomprehensibly huge; they’re just a number. There’s no sense in trying to ascertain the value of something when you have absolutely no conception of the scale at which you’re dealing with. The guy who got 2/2 right, 100%, has the right idea.
Scheduling programming, on the other hand, involves a directly tangible sense of days and months–at least, it should to an expert programmer. So it’s possible to make meaningful estimates, even if they are wrong.
Your quiz is unfair because it is designed to make people wrong. Scheduling programming is just disposed to hurting people who underestimate.
This quiz might be better if it asked for 50% confident estimates instead. Those who guess +/- infinity are actually estimating to 100% confidence, which is not the point of the quiz.
If I know Alexander’s birth-date to within a century, my idea of 90% confident is much narrower than if I only know it to within 5 centuries or have no clue at all. Because of this, the amount of knowledge you have on the subject doesn’t matter much.
90% confidence is difficult to distinguish against 100% confidence, so you get the impression that you should get them all right. But at 50% confidence, the best result is half right and half wrong, and 100% right is nearly as bad as 0 right.
That’s why I think 50% would be more meaningful, assuming I have interpreted the situation correctly.
I answer right 5. But I think Jeff is right, since my correct answers was due to my previous knowledge, and I fail almost all estimations (except for U.S. currency, with a factor of 8 range).
After know my results, I realize that I tend to narrow the range, without need it. With previous knowledge you can narrow it, but if you’re totally in uncertainy, the best practice would be to open the range and make clear that you don’t have any real data to narrow it.
As Ryan says, estimations without acuracy are useless, but so are estimations based on few data. You can estimate chances of dying in a car accident, but if you don’t know anything of the driver (frecuency of drive, drive times, country…) an acurate estimation is almost impossible.
Surely this influences the test taker to narrow their estimates? So although the conclusion that we believe that wide ranges make us appear ignorant or incompetent is correct, the test does not show this because it actually tells you not to use wide rangers.
I’m not saying the test itself is flawed but the conclusion is IMO, even if it is true. In fact it’s so true I would call it common sense, but that’s just me.
I got 4/10 btw but my estimate of 5-30 degrees for the latitude of shanghai is so close to the correct answer of 31 degrees that I’m going to say 5/10 from now on, just to avoid appearing incompetent
Very early on in my programming career (1997), I was told, “the main difference - no - the only difference between an amateur and professional is the ability to estimate the time it will take to complete a project”. By that definition (and several others), I’m still an amateur. Of course, I try not to tell that to my customer.
And that’s the problem, in my opinion, with wide estimations for consultants. When you say, “I’ll deliver what you want, and it will take anywhere between two weeks and two months”, it sends a strong message that I don’t know what I’m doing. That I’m an amateur. I’m better off saying “three weeks, no more!”, sounding like a professional, and then extending the deadline with excuses, if need be. This sounds terrible, and it hurts the customer, but at least I got the job in the first place.
I think the best, most honest answer to the question, “How long will this project take you?” is the answer, “I don’t know, but within a week of research, I’ll be able to give you an educated guess”. Of course that week might turn into two.
I estimated 9 of 10 correct, I had no basis for the blue whale question and foolishly picked too low [40-140tons], and I knew some of the questions directly [the temp of the sun, alexander’s birth, lat of shanghai].
I think what Mr. McConnell is trying to say with the higher accuracy crowd being too narrow is that we feel we missed the question we missed because the answer was slightly outside of our range. Given the methodology on the previous questions, I felt my range was large [almost 400% my low], however on most other questions, my range high was always single or double digit percentages of my range low.
A narrow estimation range does not make my estimation skills wrong, it just means I have a lot more basis of expertese [or none at all] to answer the questions given with a high degree of accuracy.
Estimations without care for accuracy are useless. [ie: the time until the sun goes supernova, % chance of an asteroid hitting the earth, chances of dying in a fatal car accident]
I enjoyed this exercise, thank you. I took a peek at the findings in terms of the poor accuracy of estimates before taking the quiz (without seeing the answers). I got 8 out of 10 right (but probably would have gotten much less if I hadn’t been clued in to widening my estimates).
I agree with most of the authors conclusions about our tendencies and appreciate that he is encouraging us to acknowledge the uncertainty in our estimates and endeavoring to give a wide enough range to make then accurate instead of arbitrary. The only thing I didn’t agree with was some of the emphasis placed on no one getting 10 out of 10 correct. While taking the test, I continually tasked myself with being 90% “sure” of my answers. This should follow to establishing an average of 9 out of 10 answers correct. Yes, some probably should have gotten 10 correct, but that was not actually the goal. The goal was to be 90% sure, and some variation either way is understandable. Even I would have been capable of getting 10 our of 10 right with broad enough ranges (answer infinity for everything!!!), but all things considered, I am not particularly disappointed with my ultimate performance.
Some of these comments are interesting, but unfortunately many of them assume that my book says things it doesn’t say. The quiz is early in the book, and it is intended to generate interest in the rest of the estimation topic. I believe every objection and reservation in these comments is covered specifically in the book.
Most of this discussion is from 2006, when the book came out. I imagine quite a few people today would say, “We’re doing Agile development, so estimation doesn’t matter any more.” What my company has been seeing recently, however, is a pendulum swing back to including estimates in Agile environments. The good news on that pendulum swing is that some Agile development practices provide an excellent foundation for accurate estimates, better than we ever had with waterfall.
