To Hendra Saputra:
It’s a hard, long road. “Fearless Change” by Mary Lynn Manns and Linda Rising is a good place to start 
Best of luck.
Parkinson’s Law is not a Law. It has been demonstrated from extensive studies (for example, read PeopleWare), that the highest productivity comes from a lack of deadlines. There are numerous theories as to why this is the case. The next best productivity rate is a when a deadline is set by a third party guru (not your supervisor, not yourself, not you + your supervisor).
Somebody said this (or something close to it) a few topics ago: “Why do development discussions always polarize to the extremes”.
The problem I see here is that people are saying “BDUF didn’t work on project X therefore no design is preferable”. Or they are saying that “Waterfall, a rigid weakly iterative approach, doesn’t work therefore we must have no process unless it is highly iterative (and of course, the most iterative approach is to hack)”.
These arguments are completely flawed since they lack any kind of logical support.
I am a proponent of design. I believe that very few software projects do anywhere near enough design. In other words, I am adamantly opposed to the “vague idea forms; hack it out” approach. However, I am not a proponent of waterfall - I do not believe that one can or should completely design a system up front without some actual coding/experimentation. This is a middle ground attitude - not a polarization or extreme attitude. Although I regularly attend the Agile meetings in my home town and have listened to and spoken with Agilists such as Alistair Cockburn, Mary Poppendieck, Steve Adolph and Phillipe Kruchten, I’m not even convinced Agile is the correct route. One problem with Agile approaches is that every last one of them gets somehow corrupted by developers to mean “vague idea forms; hack it out”. This is what happened to XP, it is what happened to Scrum at the last software company I worked for, etc.
The enemy is NOT waterfall, it is NOT BDUF, it is not even BHWNP (Blind Hacking With No Plan) it is those who would drive any reasonable iterative process out of balance toward one of the extremes because they don’t understand the needs of others on the project. Business people, managers, customers need vision into the project to verify their requirements are being met - C# code does NOT provide that vision. Developers need to verify that ideas they have in mind will work and need to prototype. The middle ground that ALL should be happy with is the various layers of design so well described in the UML and in Cockburn’s Writing Effective Use Cases.
Although it is not fair to condemn Steve Yegge’s comments based on only the knowledge gleaned from Jeff Atwood’s initial post here, I would suggest that Yegge is claiming Agile to be too much process - he is taking an extreme position of “let the developers discover cool project ideas without business/management interference”. This is likely the exact reason that Google has only one highly recognizable/differentiated product - the search engine that was driven by a clear business need so long ago.
While it’s probably not relevant to the discussion, Tom S., that site is pure genius.
Johan Tibell wrote
Why choose Enron? There are plenty of companies with a similar strategy.
Did you read the linked articles? The comparison to Enron was that Steve Yegge’s descriptions of the culture at Google are eerily similar to the descriptions in a 2002 article of the culture at Enron.
Neil gives a good example of other similarities in his comment.
Woh, way too many comments to digest, so I will just say what I would have said if I had been the first to comment… we are talking methodologies here, and there have been many in the past, many in use now, and more to come… I have personally worked using upwards of 20 different methodologies or variants over the years, have lived through introductions of new methodologies several times, and led a project myself to implement a new methodology… the issue is not which methodology is the best, it is which one is best for you, your project, your company. If Waterfall or RUP or IEM or whatever is working for you, don’t change to a new methodology just to be trendy… if your current methodology is NOT working for you, then turf it for sure, but be aware that this takes some time… The point of a methodology, in the end, is not control or command or standards or rules, its about communication, within the team, out to customers, and on; a methodology gives a set of terms, methods and guides so that everyone on the project knows what it means when you talk about Requirements, or Design docs, or Specifications, or Phases, or Iteration, or Prototype, and on and on.
How else can I say this? If Waterfall is in many peoples’ view the equivalent of Mac Truck, and your project just needs a VW Bug, don’t buy a Mac Truck… but sometimes you need a Mac Truck…
I think it’s tempting to fall into what Eugene Volokh called the "a href=“http://www.volokh.com/posts/1133568627.shtml"reverse Mussolini fallacy/a.” – that since Mussolini made the trains run on time, and Mussolini was bad, then making the trains run on time is bad.
In this case, Enron had a star culture, Enron was bad, ergo star cultures are bad.
But I think the thing about Enron was that the star culture was built around perceived talent and potential as manifested by degrees. It seems to me that Google, and the software industry in general, measures stars by what they’ve actually accomplished, rather than what diplomas are hanging in their office.
Mr. Wright is definitely Mr. Right this time.
Whether BDUF, waterfall, agile, or foobar is best depends on external circumstances.
If you’re working internally directly with the direct users (or a group who can officially take the rap), then Agile can work. Not “only will work” or “must work”, but “can actually be done effectively”.
Otherwise, with any other user type, it can’t work. With any other user type, there has to be a paper trail of culpability. Has to be, alas. Note that Joel’s business fits this circumstance.
The other externality that really matters is the nature of the application. If you’re doing something that’s amenable to taking some input, storing it, and regurgitating it in some form; then you’ll make a better application (no matter the development paradigm) if you take database design seriously. If you’re doing a data based application and take the attitude that any knucklehead coder can specify a bunch of SQL tables; well, you’ll fail. This can be done iteratively, and so on. If data isn’t the issue (you’re doing something that really is all code, like a device driver), then do what you want. Again, the paradigm is secondary.
And, lastly, there will always be iteration, if only fixing bugs. It’s merely a question of local feedback vs. global feedback (for those with an electronics bent).
You see, I don’t get to work with talented people.
So, most unit tests I’ve seen are the worst form of cherry-picking, and are used as an excuse to ignore reports of logic errors.
Most iterations are just the regular old “death march” in fast forward, and result in nothing ever getting merged, even if it’s important.
Pair programming is social terrorism; if your colleague is too shallow to even try to understand your code, it’s “stinky”, even if you’re right and can prove it.
The items on the index cards have meaning only to a tiny minority of the team; attempts to refactor and simplify can and will be shot down because they might require a reorganization of work units.
Agile in the hands of idiots is worse than Waterfall. Much, much worse.
“You folks need to get state of the art. If you’d been to Waterfall 2006 you’d be singing a different tune.”
I do look forward to Waterfall 2007. The theme of “Over the falls chained inside a barrel” just grabs you and tosses you ever the edge.
Whew! Seems like a lotta Agile bashing going on lately. I can’t help but to wonder whether those who bash it intensely have practiced it in earnest. Ron Jeffries’ “Extreme Programming Adventures in C#” gives a great insight into just how it can work.
I don’t practice it currently, but have, and know it does work when done properly.
Someone commented, “Look at other mature engineering disciplines and learn from them; software development really isn’t that unique.” Well I just happened to be reading about Denver’s massive transportation project (trexproject.com), and it sure sounds alot like agile; “… design-build… allows the project to begin construction while completing design”.
Nonetheless, I agree with Stuart; “Do as much design as is needed to get you started.”
Good post. I read Steve’s post last week and I have to agree. It’s more a rant about how great Google is as opposed to how bad agile is. I mostly read it for the Google tid-bits.
Too much of anything is bad for you… agile or waterfall.
Well, I liked Steve’s article even if it was a bit one-sided. Having said that I rarely see pragmatic Agilists writing, I only ever seem to encounter the [apparently] rabid sort. I suspect this is because the pragmatists are just getting on with things and leaving everyone to work things out for themselves.
I don’t believe in BUFD and have never worked in a Waterfall environment even though many of the other engineering disciplines around me seem to think they do. The hard fact is that whenever a design activity happens, Waterfall goes out the window. Having said that, once you commit a electronic circuit to PCB you don’t have all that much opportunity for refactoring your design. The only tools you have to you are the ability to break a circuit with a knife and the ability to solder wires directly onto pins which should have been connected to a component but weren’t because you screwed up the net names on your schematic and you didn’t pick it up in testing.
Having said that, once you get to rev 3 of your PCB, it looks pretty good. That doesn’t stop mgmt from wanting rev 1 to be perfect.
It does show, however, that design is hard and even with the greatest tools in the world you can still make simple mistakes.
So what has this got to do with Agile?
A little.
If Agile is about people and relationships I am all for it. If it is about some mystical process which I have to breathe in and become converted to, I think I’ll pass.
I have managed to avoid most religions up to now. Some might consider by use of tabs/space and my (rather unique) alignment policy to be bordering on religion but I do not; at least not until I write a book about it!
Personally, I am all for adaptability. I have never seen a specification set in stone or even concrete. I expect it all to change.
Just don’t expect me to keep to my original schedule if you change what you want.
And even then … watch out! I am notoriously bad at estimating things I haven’t done before.
Unfortunately, I live in the world of the schedule. We need to bring something to market by X because that is when the trade show is. And it is one of those every second year shows so it is big and important. Well, X means fininshing by W which means feature-complete by V. That’s only a few months away. We’ll hardly be getting started.
So let’s talk about reality.
You don’t really want an estimate, do you? You want me to hit a target.
So lets talk about scope.
OK - I guess I feel compelled to defend BDUF - in certain circumstances. The company at which I work produces software for safety critical embedded systems. We have verification teams separated from design/implementation teams, not merely because we want to, but because we have to - you need to ensure independence of verification from implementation. So…how are you going to communicate the intent of a piece of software without a design? Look at the code? Don’t think so. In addition, when you have to demonstrate traceability from customer requirements down to code, you need intermediate forms to facilitate that.
Still, we do do iterative delivery (always have done, really), so that makes us agile-ish
And we’re big on unit testing - so much so that a Paul Hogan paraphrase is needed when looking at TDD - ‘you call that unit testing’?
And when I write non safety critical software (which is most of the time), BDUF goes out the window. Do as much design as is needed to get you started. Leave it that.
Couple of points here:
-
Joel, Steve, Microsoft, and Google are right: talent matters huge, and trying to disprove decades of evidence in several industries by citing one counterexample - Enron - is a pathetic argument. It’s pure Argumentum ad Hitlerium. Enron didn’t fail BECAUSE they hired talented people, it failed because (a) it was thoroughly corrupt at the upper levels, and (b) it used MBAs as the sole indicator of talent.
-
I personally don’t subscribe to the BDUF or the NDUF (No Design Up Front) methodologies; I am a firm believer in KHTDYFJ (Know How To Do Your F$cking Job).
Any developer worth his salt knows that the right amount of up-front design (as well as the right amount of up-front specs, prototyping, architecture, etc.) depends very heavily on context.
It’s so easy to claim that XP/Scrum/Agile/etc. is “still better than the waterfall” because:
[a] Programmers get to do more of what they like (code) and less of what they hate (plan / communicate);
[b] One catastrophic failure can be disguised as hundreds of smaller “enhancement requests”; and
[c] The constant motion (“release cycle”) proves that things are getting done, which they are, except that they should have been done months ago.
Agile is merely a nuisance if practiced by talented programmers with good managers. It’s a disaster when used by average developers and profit-hungry execs. Either way, it’s much, much worse than the waterfall. It seems to succeed in the short term, just barely, but hurts much more in the long term. I say “Anything but Agile”, even if that means waterfall, but like I said before, I’ll take KHTDYFJ any day.
Sometimes, you ARE gonna need it, and if you lack the ability to think ahead, your code is eventually going to end up on The Daily WTF.
The waterfall, although it has flaws, was used for many years for a reason. It works. Agile is the new thing, and it works too, but differently. I think that we need to look at both and really find out WHY each one works to get a real improvement.
Waterfall works becuase of a few things:
- You know where you are going before you start
- Seperation of responsibilities has some advantages (at least between developer and tester)
Agile works because:
- Requirements change, agile handles it.
My theory is a little different.
- The designer, developer and business analyst should all be the same person. If you don’t understand a problem, you won’t make a very good solution.
- The tester, requirements provider, and end user should be the same person. Who better to test whether it met the requirements.
- Design is upfront, but from a what to do, not a how to do it standpoint. (Requirements and design are really just the developer discussing how technology can fit the business need with the user)
- Parts are small enough to be completed quickly so that they can be tested before the whole thing heads off in the wrong direction.
Keep the user involved so that as things change, they can be handled (agile) and keep the parts small enough that the change is not catastrophic (agile) but know where you are going on each deliverable before you start it, and it is not done until the tester (user) is happy with it (waterfall). Knowing what you are going to do, and what the project should accomplish is the only way to set reasonable delivery dates or costs (waterfall)
-
Almost every tool the agile programmer uses was developed with waterfall. Waterfall therefore can’t be the almost guaranteed failure they insist it is.
-
Most agile arguments are made against a caricature of waterfall that even places like IBM didn’t practice. The author of this post makes that same mistake.
I’m at a loss to understand your “anything but Waterfall” mentality. Waterfall seems to be the only programming “methodology” [I hate the word] that pays anything more than lip service to the idea that software, like any other complex product, needs to be DESIGNED. And in my 34 years of computing experience, I’ve found that nothing leads to more productivity and responsiveness in a software project than having great design. Conversely, nothing slows down an IT department’s response rate than having to deal constantly with the instability, bugs, and other difficulties that are inherent in poorly-designed or non-designed software.
For many years I successfully delivered very complex systems using a practice that I call “schedule-driven Waterfall”. Every quarter the business agrees to the requirements for the coming quarter’s development. After locking in something that can be reasonably designed and implemented in 3 months, developers proceed to build what was specified, with the first month of each quarter dedicated to pure design, and with no need to be shooting at a moving target. The business (or the client) may request changes at any time, but it is understood up front that no such “late” changes will be implemented until the following quarter at the earliest. The reward for this maturity? Quarter after quarter my clients got something that I’ve never seen delivered by any project using any other methodolgy: a thoroughly tested product with zero known bugs, and a near-zero bug incidence thereafter.
This process works but it requires discipline on the part of the business/client - i.e. the understanding that you simply cannot conduct a successful software development in an environment of continually changing requirements. It is an adult, real-world view based upon real-world limitations on what can actaully be done, and it requires the presence of grownups on both sides of the IT table. Which is perhaps why it will never be the fad du jour.
I read the Agile Manifesto, which supposedly defines the Agile mindeset, and it seems like some sort of back-to-the-60s hippie thing. “Individuals and interactions over processes and tools?” “Responding to change over following a plan?” “Customer collaboration over contract negotiation?” Yes and we’re all going to have peace and love and get ourselves back to the Garden just because we want to. Right.
Sorry, I’ve been there, done that. Any project that isn’t sitting on top of a great design and whose programmers aren’t obsessively and constantly concerned with delivering great design is doomed to an eternity of low productivity.