I Pity The Fool Who Doesn't Write Unit Tests

blog
21

“Sadly, it’s the “senior” developers that are often most resistent to trying new techniques like TDD - years of doing things a certain has them set in their ways.”

I have an alternate explanation, one I think is much more credible than “old fogies don’t know what’s good for them”. No, it’s that TDD is a teaching tool which benefits beginners much more than it does seasoned veterans. Junior programmers are more likely to get fanatical about TDD because it actually did help them advance the learning curve from complete novice to journeyman programmer. They also likely did it on projects that were easily amenable to TDD, such as college projects or new from-scratch development.

And senior programmers are more likely to say, “huh, this is a practice which will teach me me things I’ve learned long ago and which will force me to program the way I always have. Alright, so why should I bother?”

Sure, you CAN shoehorn any sort of development in the TDD model. So what? It’s still shoehorned in. You don’t get the same benefit that you would, say, with your legendary suite of a zillion tests, each which run in under a millisecond, which cover every aspect of the code in complete isolation from each other. Instead you have something approaching the results of traditional QA, where you have limited tests that try to test things in isolation but usually don’t, with big gaps in coverage and still don’t catch the complex interaction bugs.

You know, the choice is not between “no tests and some tests”. Whenever someone says that I want to slap them. Traditional QA is not about writing no tests. Traditional Dev is not about never writing an opportunistic test when they feel like it. The choice is between some tests and more tests. Are more tests always better? Hell no! I’d rather have no tests than a suite of fragile tests which never catch anything and throw off hundreds of false positives. Granted, more tests are USUALLY better, but you know, those “more tests” don’t come for free. I may WANT to write more tests but choose not to for the same reason I may WANT to buy a Ferrari but I end up buying a Honda Civic – because it’s the best value, and my time and money is better spent elsewhere.

Okay, to sum up. I’m not saying “TDD doesn’t work”. Clearly it does work very well for some people and some projects. What I am saying is that TDD doesn’t work equally well for ALL people and ALL projects. For some combinations of people projects, it’s not worth it for them to go through the exercise. And those people can do less with the wet-behind-the-ears TDD zealots with their condescension, their pity, their starry-eyed idealism and their “if you’ve never tasted the kool-aid you’ll never know what it’s like”.

22

One could claim TDD to be our latest attempt to get written specification of what the customer wants. The bugs I run into are mostly the result of late design changes/feature additions rather then bad design/coding.
With TDD you get a sort of a contract (think interfaces here) between you and you customer (this is even more important in in-house projects). This assures that the deliverables are specified and that you are actally building something someone wants/needs. This eliminates the “Oh, how about we add this…” ideas at project progress meetings.

23

Foobar, if you think J. Miller article useless, what you sugest then?

There’s nothing really out there right now that truly helps with comprehensively testing web apps. I’d say we’re still about three-four years away before we get something that’ll be ready for prime time.

But for now, NUnitASP would be better for testing, especially for existing applications. Even then, you’ll still have to do a lot of the usual load-and-try-out-the-page testing.

24

I am 100% with Alyosha. “There is no silver bullet” can not be said often enough. I have seen unit testing work pretty well on front-end simple stuff you just have to write. But in every case that is more research than just development they are just bullshit. So, yes, you want your junior programmers to write unit tests, but that’s it.

25

Reading your topic about Unit Testing, I ask: Do you unit test asp.net applications? Do you use NUnitAsp? Or what?

I was late to the unit test party, so only my “latest” codeproject article has unit tests-- the one on encryption. And the unit tests helped immensely! I not only changed the API based on my external use of it, but I found a few bugs as well.

unit tests are so important that they should be a first-class language construct.

Meaning, the unit test attributes should be built into the .NET framework itself, and you should be able to run tests as part of a command-line compile cycle. And the Visual Studio IDE should support TestRunner type functionality out of the box.

Even then, you’ll still have to do a lot of the usual load-and-try-out-the-page testing

You don’t like the watir thing? It drives IE to run the tests, so Javascript, etc, is evaluated.

a href="http://wtr.rubyforge.org/"http://wtr.rubyforge.org//a

Seems like a good idea to me.

26

Hmm, I find the main of advantage of writing the unit tests before writing the code is that it helps me concentrate on how the unit will be used, rather than how it works. That pretty much matches point 6, though it’s less to do with the units overall design and more to do with its interface. In the end design is something that you do in your head; and only experience and better mental models are going to improve that.

The other benefit I find of writing tests first is that it acts much the same as a detailed to-do list. Giving me a sense of accomplishment when I tick of a task/test, and guiding my activities.

Other than that the most of the points in the article simply spell out the benefits of unit testing in general, not necessarily test first development, which makes the headline “Twelve Benefits of Writing Unit Tests First” a bit of a misnomer. In fact the article seems to point out the opposite: That the benefit between writing the tests before vs. after writing the code isn’t that great.

27

Thank you Brian, I will check about Model View Presenter.

Foobar, if you think J. Miller article useless, what you sugest then?

28

Jeff,

What does this mean ?

“… that unit tests are so important that they should be a first-class language construct.”

Regards,

Justin

29

This point has been made twice, and I don’t understand it, so I MUST be missing something:

"Most of my bugs come from boundaries. Code that interacts with…
the database.
the file system.
the GUI.

Unit tests simply don’t address those concerns."

These are all just sources of input, right? Sure they’re variable, but that’s what most unit tests are designed to test – variable inputs.

Beyond that, the file system and DB are transparent to the testing environment through IO objects and SQL. Can somebody give me a concrete example of a DB, GUI or filesystem input/scenario that isn’t testable?

30

Dave, I think you’re using the term “unit testing” to refer to all kinds of testing. It’s not – there are tons of different testing techniques besides unit testing. Yes, unit testing tests variation of inputs, and traditional QA testing tests variation of inputs, and the difference between the two is a matter of scale.

Remember, received wisdom about unit tests is that they must execute QUICKLY – or else your devs will get too bogged down running all the tests every time they go through a red-green-refactor cycle. Executing quickly means avoiding lengthy setup procedures and it means testing things in isolation from each other.

With network and/or database dependencies, you are essentially testing large, complex data sets. You will naturally get large, complex results back. Not only are your setup/teardown costs going to be much larger, but you’re also not testing things in isolation – it’s harder to enumerate all the edge cases and provide good coverage.

It is bad testing practice to verify that an API call results in such-and-such database modifications and such-and-such XML being sent over the network. Because that’s testing implementation details, details which are the first things to break once any refactoring happens. You should test what goes IN to an API and what comes OUT of it only.

31

No, they are not “just sources of input”.

They are also destinations for output.
Databases can perform a lot of processing unto themselves. This cannot be stubbed-out, as the error may be in the database trigger or stored procedure.

Any time you touch data in a file or database, you run the risk that another application is going to also touch that data. That means you need to not only test your application, but also any application that uses the same data.

One has to also consider the difficulty in testing bad data in a database. Unlike a traditional unit test, where you can literally pass in anything you can code, a database needs to be pre-populated with both the good and bad data you need to test. This cannot be simply tossed in the build like a unit test. It needs to be carefully planned for.

32

Does anybody have some good links that show benefits/drawbacks of unit testing in web (browser) focused development.

I am coming from a typical LAMP environment and develop intrant applications in a small team of developers (usually 3-5).

Usually the majority of the bugs we face or related to the user interface (typically compatibility issues with browsers or bugs in complex client side scripts) and therefore i don’t see a good use for TDD? Am i wrong? Are there automation tools for browsers and/or JavaScript engines? Or is TDD not suitable for UI focused problems at all?

Running static tests in a “perfect world” of a testcase simply does not match the stupid reality of so many bad and full of bugs implementations in browsers.

33

Boy I love reading the self righteous “I’m better than any test” posts. Clearly an experienced programmer won’t get as much bang for the buck as a less experienced programmer will from units test… up front.

I’m going to assume that even experienced programmers are still not writing perfect towers of code the first time with perfect execution of every use case… since that is impossible in the real world with the realities of moving business requirements. Yet, that is exactly what we are being told by the “I’m better than any test” crowd: that they don’t need unit test because they already are doing what the unit tests would force them to. They never write bugs. They never refactor. They write gleaming ivory towers of code that are unassailable by error, user requirements or time.

Hogwash. Anyone who told me they could do that would be shown the door quite quickly during the interview process, because anyone who touches real code knows that such a perfect utopia doesn’t exist.

So what do we gain by tests? We gain the ability to refactor, with confidence. We gain the ability to ensure that error we fixed doesn’t crop back up when some other coder touches the code and doesn’t realize that strange case actually had a use. We gain the ability to give code to a mere mortal programmer and have some confidence that when they are done with it we know what damage was done (assuming that things didn’t go well) and that we can go back to a known good CVS state… or we can be assured that at least those cases that are tested work.

In other words, tests provide those who live in the dingy, complex real world can write code with similar confidence to those who live in imaginary ivory towers were requirements never change and nobody ever writes buggy code.

34

So, where does one get started in the whole Unit Test thing?

I use VS 2003 right now. Should be adding 2005 soon. But I also use a lot of VBA.

And, I am a solo developer with no budget for cool tools (even for the really necessary ones :frowning: ).

35

“Hogwash. Anyone who told me they could do that would be shown the door quite quickly during the interview process, because anyone who touches real code knows that such a perfect utopia doesn’t exist.”

Beat up straw men much?

No, the utopia you describe doesn’t exist. But it’s also not the logical conclusion of “I try to get it right the first time”. Design mistakes can and will be made by even the best programmers. The process should accomodate that fact without explicitly optimizing for it. I would be suspicious working in a place where refactoring the world is the norm because of all these wonderful unit tests make the cost of change almost free. That strikes me as a very amateurish way of working: churn out anything that works, we can fix it later.

It seems some people have this deep-seated disbelief that people can get better at programming with experience. Sure, you can always point to some 15-year old fogie who churns out crap day in and day out. I’ve known plenty of programmers like that. But I can also point out a few 15-year old senior devs that have good design sense, remarkably high productivity and remarkably low bug rates – people who generally DO get it right the first time.

I admire how well you can identify the benefits of a good test suite. I agree with most of them. What I think you’re a little blind to (and folks have been consistently missing this point I’ve been making) is that there are costs in writing such a test suite too. And sometimes tests emit false positives which can be another huge time sink. My point is that sometimes the benefits outweight the costs, and sometimes they don’t. One size does NOT fit all. Unit tests do NOT solve all problems. We should have the freedom of saying that without being slapped with this “elitist anti-TDD crap-churning old fogie” label.

36

Every anti-TDD zealot I’ve run across invariably has yet to get over the learning curve, the logistics of implementing and automating unit tests, and are either too lazy or too arrogant to do so. Once JUnit or NUnit is set up, the overhead of writing a new test is minimal. The “waste of time” argument is ridiculous – the vast majority of unit tests I write take seconds to set up and run, and save me massive amounts of time I would spend otherwise mucking around in the GUI and debugger, setting up watch variables, etc. I can harness up the method in question so much quicker with a unit test. So its benefit as a development tool alone cancels out any percieved “cost” of writing – what? – another 10-15 lines of code per method? Just how slow do you people type? Give me a break.

Of course, excecution always trumps concept, and if you’re not thinking through all the edge cases (yes, sometimes even changing APIs) then neither TDD nor anything else can help you.

37

To Aaron’s points:

  1. Excecution trumps concept. Unit tests only automate and expidite what a good coder should be doing anyway, i.e., thinking through inputs/outputs and considering all the edge cases. I have yet to come across a cases that I couldn’t test for.

  2. No. If you integrate unit tests into your daily workflow and coding habits, it’s actually faster and at worse a wash on the really difficult cases. There’s probably a library out there to automate for your environment (JUnit, NUnit).

  3. Yes, there are some things Unit Tests can’t do.

  4. ??? Exactly what DB operations can you not test? Run a query. Make an assertion.

The app I write tests for is a front end for a Database. All I do all day is write code for a database app. The thought has never crossed my mind that “unit tests are mediocre for anything database-driven.” I guess I just don’t understand your point.

  1. (see 3) Unit Tests don’t design your code for you. They can’t tell management to give you better requirements, and they won’t wash your socks.

@#$%*$ Unit Tests!!

  1. I’m not following you here either. Write good MVC (or some variation) code and think through the edge cases for user input. Test the controller. Isolate the UI from the business logic. Test the model. I have yet to think of possible user input that I couldn’t test against.
38

One last point to Aaron:

Size truly doesn’t matter. A class is a class. A method is a method. They test the same whether they’re in a package with 2 other classes or 100.

The database app I’ve worked on for eight years – with tons of ugly legacy code that isn’t unit tested – is approaching a million lines, eight to ten developers committing to source control daily.

39

Aaron “Code reviews and -actual- testing (by a good tester) will reveal deficiencies much faster and with better accuracy.”

This has been the opposite from my experience. It is SO much more expensive, process-wise, to have a tester or code reviewer catch a bug. If I can turn in solid, tested code the first time, that’s a win, hands-down. The coders I know who unit test do this much more cosistently than those who don’t, and they develop a reputation for doing so.

We’ve tried a “hybrid” approach for several years now, where unit tests were considered a good idea, but not enforced. It’s been frustrating. The anti-TDD zealots who didn’t adopt (mainly because they had such a high opinion of their own coding practices, and because they didn’t want to take the time to get over the setup/learning curve) spent a lot of time complaining when they’d break somebody else’s test; complaining that they’d rather be “actually coding,” etc. Oh yeah, and they spent a lot of time quoting James Bach to anyone who would listen.

Over time, those coders developed a poor reputation with QA, and the “unit testing” crowd began to be held in high regard. We’re still a “hybrid” dept. in theory, but – funny thing, this – a number of the non-testing coders aren’t here any more, and most everybody else pretty much writes test for their code, not because they’re forced to, but because it works.

40

You don’t like the watir thing? It drives IE to run the tests, so Javascript, etc, is evaluated.

Looks good, but it’s still quite limited. The main problem with a web page is that even though HTML/CSS/etc on the surface looks easy, creating a layout and adhering to that layout is damned hard. That’s why you still need to look at the thing to see what’s been bungled.

As for Aaron’s comments, creating unit tests is actually pretty easy. Oh well.

What’s really funny is that there’s two arguments going on here: The merits of TDD and the merits of unit testing.

TDD is merely another variant of the SDLC. Big whoop. Use it or not. I’m not convinced it’s any better or worse than all the other gazillion SDLC variants out there.

Unit testing, on the other hand, should be mandatory. It’s really the only way we have right now of formalizing, capturing and persisting testing procedures. What many people seem to have a hard time with is that you’re actually creating another client for your API. Here’s the catch. Most programmers, who are actually damned hacks, couldn’t write a business interface layer to save their lives. For them, unit testing will be very hard, what with business logic strewn all over windows forms, web forms, http modules, event delegates, App_Code, servlets, JSPs, etc. So for them, writing a “client” for their non-reusable code is impossible, even from a conceptual viewpoint. Even for those of you who do work on project where you were fortunate enough to write a business API, I’d bet most of your co-workers don’t even realize they’re using an API.