The Ultimate Unit Test Failure

We programmers are obsessive by nature. But I often get frustrated with the depth of our obsession over things like code coverage. Unit testing and code coverage are good things. But perfectly executed code coverage doesn't mean users will use your program. Or that it's even worth using in the first place. When users can't figure out how to use your app, when users pass over your app in favor of something easier or simpler to use, that's the ultimate unit test failure. That's the problem you should be trying to solve.


This is a companion discussion topic for the original blog entry at: http://www.codinghorror.com/blog/2008/02/the-ultimate-unit-test-failure.html

unit testing is used for checking code quality: to see if waht you’ve written is well enough written. Does’nt tell you what to write. Does’nt tell you if the code is useful. Does’nt tell you anything about your users, like the compiler that does not tell you how many copies of the software will be sold, and the version control system does’nt know if your icons and splashscreen are nice.

I always thought it was “State Puff” marshmallow man. That always sounded more scary.

(Programmers can be tangential as well).

Hey now Jeff,

This is a bad and confusing post. I don’t understand why you are talking about this.

Kris Katto

sounds like Jeff is having a bad test day :slight_smile:

I generally find unit tests and code coverage a bit over rated. Code coverage for core classes in a system is vital. But unit tests for fringe components like UI classes, xml configurations, data…? give me a break.

That said unit tests and UI design/interaction are orthogonal concerns. The post itself is not completely valid. If it had been importance of UI interaction/design versus system architecture, then I would say it makes a lot of sense. Many projects/products fail due to over zealous architects wasting time and resource on ‘cool’ system internal designs and implementations, while completely ignoring the UI.

Breath through your mouth. No! Breath through your nose. Arrg your doing it wrong!!

I dig where Jeff is coming from here, let me help him a bit :slight_smile:

Engineering typically obsesses (and arguably rightly so) about the engineering effort. We focus on the building a robust, extensible system with lots of moving parts - ensuring the quality of said parts through significant engineering effort.

What “we” (the royal we - “the editorial”) sometimes fail to do is understand that what and how we build should change when faced with expertise telling us what to build.

For example:

“We need to rework the reports part of our app now that we have this acceptance data telling us we got it wrong.”

“But we won’t be able to get the code coverage up to our required 75% by release time! Forget it!”

The “for engineering” part of that response fails to take into account is that code coverage on a crap system doesn’t give you anything more than… code coverage.

I’m just impressed that Cooper can throw out a Dune reference like that and then switch gears in mid-argument to Ghostbusters and have it still make sense. Slick.

Agreed. The first thing that came to my mind was verification versus validation. While Engineers should be concerned with whether they are building the product right (verification), their overriding concern should be whether they are building the right product (validation).

I kind of think that one of the only things that separates one software company from another is mastery of a particular problem or user domain. Our company, Primordial, is always trying to master the military user domain. We keep a risk database for each of our projects to facilitate risk mitigation analysis, and the risk that “users won’t find our software useful” is always near the top.

It is important for engineers to get their head above the water and see the bigger picture sometimes.

Unit testing should be separated from User Acceptance testing; that is, your Unit Test should never drive the acceptance of the user interface as by their nature, Unit Tests do not and should not have the entire execution “thread” for the process that the app is implementing.

Sometimes the developers have to get out of the way.

The sad part is when the specs aren’t coming from experts they are coming out of some managers head and you get no idea what the customers really want. Then its Dilbert time and the product is likely to be crappola even if it does have unit tests or not.

We may have all been marshmallow men at some stage but sometimes its justified, sometimes its not.

A gree with the sentiment that this post is confusingly titled and confusingly written.

the ultimate unit test is whether or not users want to use your application.

Absolutely not. That’s not a unit test at all, since it is a test of the entire application not a discrete unit.

It’s a User Acceptance Test; and, as you rightly point out, it’s the most important one of all.

“Unit Tests do not and should not have the entire execution “thread””

So very true. Unit tests are not an endpoint, just one tool in a suite of many required to ensure your product, well, works.

Where “works”, of course, means “correctly does the tasks expected of it in a manner that makes people willing to use it.”

I thought the post was right on-target. Seems like some of the comments are by some of those “obsessive by nature” people who can’t see the forest through the trees.

If you user interface sucks or is even remotely difficult to use or understand, you’d better hope you’re the only kid on the block. Because users won’t spend more than a few minutes with a piece of software that they can’t navigate no matter how many unit tests you ran before you published it.

Jeff, I’m sorry, but I just don’t see how these two things are in any way exclusive. A [well-tested unusable application] and a [under-tested highly-usable application] are NOT two ends of a 1-dimensional spectrum.

Too often, I find myself working on projects where the visual style and to some extent the GUI mechanics are viewed as something that can be determined “later”, after the “engine” is written. When later comes, there is inevitably some major change that needs to be made that reaches several layers down into the system. It’s like I’m being asked to remove raisins from cookies after the dough is mixed.

I strive to deal with this gracefully, but I have to ask: is it any wonder that the average engineer tries to defend their project against seemingly-whimsical changes imposed by some distant self-styled authority? Respect comes from trust which is not gained by dictatorial power (which appears to be the solution put forward by your quotes). Rather, it grows from experience of the designer’s competence, which never happens in a development process where engineers are isolated from users by designers and management.

This post could be replaced with:
“Make your applications simple to use and easy to understand.”

Jeff, You should unit test this post for usability.

His book „Inmates are Running the Asylum” has changed my way of thinking about UI and interaction design.

Hey Now Jeff,
Nice point, we should provide solutions.
Coding Horror Fan,
Catto