I Pity The Fool Who Doesn't Write Unit Tests

blog
51

Aaron wrote:

“If design problems account for 80% of the debugging (I’d say that’s a conservative estimate) then even if unit tests halved the time spent on code problems, it’s still only saved you 10%, and that may very well be offset by the time spent creating and maintaining the unit tests.”

I’ve never seen design cover 80% of the bugs. Maybe you work in a department of 20-year veterans with immaculate coding practices, but in my world there are a lot of recent BS and young coders still working it out. But even if I were in a deptartment with a bunch of ubercodemensches, there are still things design couldn’t do.

Brilliant design cannot write methods. You cannot design null checks. Design cannot guarantee that when other code monkeys break my code, I’ll know about it within the hour. Our dept. does a lot of design, QA and code review. Testing is just one tool in the box, but it’s an essential tool that covers things design and code reviews cannot.

And I’ll beat this horse again – it’s faster to code with unit tests than without. It’s even faster to code good design with unit tests. There shouldn’t be a dichotomy between the two, but in my day-to-day reality, I don’t always have control over what requirements I get, or who’s in charge of the design process. But I do decide whether my code is going to be tested or not.

52

“The utopia of separating UI and business code.”

Hmm, I’m about 95-99% in utopia then.

What a naive statement. Another damned hack.

53

Going back a bit here, but the latest version of Visual Studio Team System is geared explictly for unit testing. Microsoft sees the need for testing at the code/unit level.

I have worked on more than a few functional and system test projects in QA. The worst part about testing at this level is finding a bug in production that a unit test could have caught. They exist, they are not unicorns. But what saddens me is how quickly a developer will place the blame on the QA group. We cannot test everything, unit testing is essential.

Resistance to unit testing from developers is the first part of the problem. But lack of understanding from business sponsors and business analysts alike only complicates the issue. If you deliver a poorly designed system to a developer you will get a system, but it won’t be what you really had in mind.

The politics of this makes me sick. Developers need to follow requirments, and code what is asked. The BA’s need to deliver solid requirements and design specs. The QA group needs to over see the ENTIRE process, start to end. Otherwise your just doing Quality Control, at which point you are better off not bothering to test. The QA group is always relagated to the role of “tester”, this is why bugs end up in production.

I used to be a developer, I know how painful it is to have your ability as an artist dumbed down to the science of the printing press. But it is a necessary evil.

Development groups need to be more open minded, this is why you see so much offshoring of development projects. Offshore resources will do as you say to a T. Keep in mind you will find holes all over your design when someone in New Delhi codes exactly what you sent them. Again QA has to sign off on that design before submission, but this goes back to the politics of software.

If you have ever approached a development team with the suggestion of someone else, maybe in QA, writing unit tests you know what reaction you will get. First they will ask “Why do unit testing?” Then they will ask “Who is going to write the tests?” And when you tell them QA/junior developer they tell you “They have no clue how to write a unit test, they can’t understand my code, and they will ask me questions all the time, how do you expect me to get anything done?”

At the end of the day any company that is finding problems in their production apps needs to look at their SDLC. The groups that participate in the lifecycle need to have clear definion of their roles and clearly documented expectations. The groups need to have a HEALTHY amount of dependency on one another. Politics need to be kept out of the process. Without this you will ALWAYS find defects in production regardless of what methodology or “religon” you follow.

54

“One day, some clueless wage slave will be sitting at 3 a.m. in front of your perfact C++ code with a stream of drool escaping the corner of a slack mouth, thinking “WTF is bind2nd?””

Hahah! Nice mental image there. Don’t think I’ve ever used bind2nd, but point understood …

I think if your coworkers really are total monkeys, then you’re just f*cked no matter what you do. No process or technique – not even one as wonderful as TDD – can save your ass in that situation. So for the sake of this conversation I’m assuming competent coworkers – knowing full well that it’s not always the case, but if you don’t have competent coworkers then any process improvement is giving medicine to a corpse.

“But doing without unit tests altogether in a real-life situation seems to me to be total insanity.”

I totally agree. Scroll up, you’ll see I’ve never advocated such a thing.

“the assumption is made that doing test driven work somehow requires that you not think ahead and do the design up front”.

TDD has many arguments for it, so not all TDD advocates use TDD as an excuse to write crap and then refactor. But some do. Of those, the argument that “TDD makes refactoring easier” is the most attractive one of the lot, because refactoring is what they do ALL THE TIME. Refactoring should be much rarer than that. One need not adopt a process that optimizes for constant refactoring.

Anyways I think we’re having a Colbertesque “George Bush: great president or greatest president?” argument here. I think we all can cite specific unit tests or developer tests that have helped us greatly in the past. But there is still a lot of art to them, and there are still too many ways to write ineffective or unproductive unit tests. And, you know, people who point that out the limitations of unit testing keep getting branded as God-hating anti-TDD commie Luddites (even though everyone accepts that unit testing does have its limitations).

No religious debate here …

55

I like how it is said I’m attacking a straw man and then the assumption is made that doing test driven work somehow requires that you not think ahead and do the design up front… or even to try to get it right the first time.

I see your “straw man” and raise you a “failure to address the points” such as the reality that not only golden children will touch code and thus designing a system that will detect failures of imagination is a good idea.

56

Would Mr T write tests before he went and smashed stuff up? I dont think so.

57

Hi all,

Would Mr T write tests before he went and smashed stuff up? I dont think so. - yourMateNick on February 20

hahahahahha lol. If you really think about it, he acctually did the tests and they were called practice. If he didnt, he might just get smashed up himself.

On a smilar note, however, I do think that unit tests are kind of waste of time. But I will definatly try them to really feel what all this hub-bub is about. From what I have read and understood, it seems more like unit tests as a tests which one can do while they code any way. Debugging and hacking works the best for me up till now, and writing more code specificaly for tests for situations I KNOW for a fact that will change will eventually require change in test modules as well…then why bother?

But as I said I will have a go at them.

58

I do a lot of my low-level (as opposed to end-user-oriented) testing in the debugger. In essence, I don’t trust a piece of code until I have forced it through all possible code paths. This might sound daunting - even impossible - but it is not. With a good debugger - and I se Visual Studio which is truly excellent - you can put your code through hoops in short order. Simulate an error condition by changing the result returned by MySuperDuperDooWopADiddyFunction () to false. Use ‘set next statement’ to walk through a rarely used and difficult to reach code path. Ensure that a particular message box is displayed, at least once, to see that it looks acceptable on the screen and reads OK. Use edit-and-continue to fix a minor bug there and then and walk it through to check that it works.

The list is endless. If you don’t work this way, you are missing out.

Paul Sanders
www.alpinesoft.co.uk

59

I do a lot of my low-level (as opposed to end-user-oriented) testing in the debugger. In essence, I don’t trust a piece of code until I have forced it through all possible code paths. This might sound daunting - even impossible - but it is not. With a good debugger - and I se Visual Studio which is truly excellent - you can put your code through hoops in short order. Simulate an error condition by changing the result returned by MySuperDuperDooWopADiddyFunction () to false. Use ‘set next statement’ to walk through a rarely used and difficult to reach code path. Ensure that a particular message box is displayed, at least once, to see that it looks acceptable on the screen and reads OK. Use edit-and-continue to fix a minor bug there and then and walk it through to check that it works.

The list is endless. If you don’t work this way, you are missing out.

Paul Sanders
http://www.alpinesoft.co.uk

60

I fail at unit testing.
I can never figure out what inputs to test and how to know what the corresponding outputs should be.

61

@Zeo

On a smilar note, however, I do think that unit tests are kind of
waste of time.

Pretty much everyone does until they actually try them.

From what I have read and understood, it seems more like unit
tests as a tests which one can do while they code any way.

Yes. But unit tests are repeatable, very quickly. You stepping through a debugger is both error-prone and slow.

writing more code specificaly for tests for situations I KNOW
for a fact that will change will eventually require change
in test modules as well…then why bother?

That’s the most important reason to write unit tests. You will find that the code being tested changes much more frequently than the unit tests. Having automated tests makes it far quicker to test, and allows far more confidence in the changes you’ve made.

But as I said I will have a go at them.

Good :slight_smile: It’s the single most effective thing you can do to improve your productivity. It is much quicker to write unit tested software than not (partly because of the vastly reduced amount of bug fixing you do, partly because you tend to end up with a better, more maintainable design)

62

@Damien

TDD creates all this inertia around the initial interfaces in the
code, which start to affect the design. Or at least I think it
does, I’ve never actually tried it.

Ah, arguing from ignorance :slight_smile: The amount of debate over practices like this from people who have never actually done it is astounding :slight_smile:

who has time to try out every amazing new methodology?.

TDD is not a technology, but a way of doing things.

63

I’m with Alyosha. 100%. Unit tests are for beginners. Beyond that they’re a monumental waste of time and effort.

Actually, this comment has it 100% wrong. It’s the beginners who don’t get unit tests. They may have been coding for years, but they’re still beginners (in the Dreyfus model sense of the word).

You see this in two ways:

  1. Unit test coverage should be 100%. You hear this from recent converts mostly, but it’s not true. Unit tests are very weak in some areas. I never unit test my UI or report layouts, for example. It’s possible, but there are better tools to test those things.

Eventually, (some) people get to recognise when unit testing is appropriate and when it’s not.

  1. Unit tests are a waste of time. You hear this mostly from people who’ve never tried it, or haven’t tried it much. In fact, if you listen to people who are experienced at unit testing, they are pretty much all agreed that writing code is quicker if you unit test it, and particularly so if you write the tests first. Furthermore, the benefits accrue over the years you have to maintain the code you just wrote.

Unit testing tests your code’s behaviour explicitly with little coloured lights, and its design implicitly, because writing tests for bad architectures is hard. Write the tests first and you test your design upfront and continually.

Writing code with fewer bugs and a better design saves you time and money both initially, and over the years when maintenance is easier and more reliable.

If you are not already doing it, then adopting unit testing is the single best thing you can do to improve your productivity. It’s not a silver bullet, but it most certainly is a best practice. Just don’t try and do it for every single thing.

64

you are very intelligent guy can you teach me some thing more about civics i am a 12 year girl i am studieng in 6th std

65

“any tests are better than zero tests”

I disagree; like comments, incorrect ones are actually worse than none at all.

66

Not a fan of TDD myself. My arguments against unit testing are:

  1. Unit tests only prove that the code works for the specific inputs/outputs that the programmer has already considered. For experienced developers, virtually all bugs occur in the edge cases that aren’t covered by unit tests.

  2. Unit tests really do take a lot of time to write and maintain if done properly, and in informal projects or any project where the requirements change, it’s easy to spend over half one’s time just keeping the tests up to date. Sometimes you know that you’re going to break old code, and then in addition to changing all the old code, you now have to change all your unit tests (or just give up on the unit testing for that code, which is often what happens in practice).

  3. Unit tests do a very poor job when it comes to threading and timers, for reasons that should be self-evident.

  4. Unit tests are mediocre for anything database-driven, where often the only way to verify correctness is to use production data and verify the results (it’s possible to do this in a full-fledged test environment, but not in a unit test).

  5. Unit tests do nothing to expose architectural flaws, which are the most expensive ones to actually fix (assuming the requirements are correctly defined). Those flaws are generally discovered during the coding process, which is obviously going to move slower if one is also dealing with unit tests, and that means the time it takes to discover such defects is greater overall.

  6. In my experience, at least 1/3 of the bugs are introduced in the UI, which unit tests can never test properly. Maybe some day computers will be able to model the chaos that humans bring into the fray, but we’re not there yet.

Given time I could probably think of 4 more points, but I think that’s enough for now. :slight_smile:

My biggest beef with McConnell (Code Complete) is his religious fervour for unit tests. As other people have commented, and as James Bach has famously stated, there are no best practices, and unit tests are too often treated as a best practice. For extremely large projects that have massive APIs which are all supposed to work in deterministic ways, unit tests can be a great tool both for initial testing and regression testing. For anything smaller (I’d say anything 100,000 lines or less), less formal, data-driven, etc., they may merely create the illusion of working code and are not always such a useful practice.

I have yet to see any truly persuasive argument that unit testing is a best practice, or even a good practice, in the general case. Having said that, if they help someone write code more efficiently/accurately, then I would not by any means try to steer that person against unit testing. People should use whichever tools they find useful; I just wish they wouldn’t prescribe their elixirs to everyone. If I were managing a team, I would not try to stop any team members from using unit tests, but I would not force the team to use them either. Code reviews and -actual- testing (by a good tester) will reveal deficiencies much faster and with better accuracy.

Sorry for the rant Jeff, you’re usually spot-on (after all, that’s why I’m here) but I think this one might have missed the mark. Can’t win 'em all though!

68

Unit Tests before you write code only works if the design of the code has been done up front. Unfortunately, I think this is rarely the case. I think most developers don’t know how things are going to work out until they start writing code. Now, whether that is a good or bad thing can be debated. But its hard to write tests if no design exists.

Ideally, the design of the application would be known so that technical testers could write the test harnesses to fully test the application. Ideally, you would want a structural test of the code (unit test) followed by a functional test of the code. The structural testing could be done by something like NUnit. Functional testing is having users actually step through the system and verify functionality.

Unit tests are valuable, provided both positive and negative test cases are supplied. Many bugs can be found during unit testing. Keep in mind during unit testing you are not only testing your code, but the numerous libraries you are employing across the application as well.

I guess my only conceern is that it compiles. Believe it or not, I have had cases where checked in code did not compile, so unit testing may be asking a bit much in any shape or form.

71

Reading the subsequent comments, I think the most common response to my arguments was “yes, but so what?” and the second most common was “nuh uh!” followed by a set of further utopian best-practices (such as “isolate the UI from the business logic”).

There’s a fundamental misunderstanding of the argument here. I said from the get-go that unit tests can be useful and I don’t have anything against them. My problem is with those who:
a) Believe that they’re a legitimate substitute for actual design and documentation
b) Honestly think that they can test for every conceivable bug
c) Recommend - nae, INSIST on it as a “best practice”.

Each of these lines of reasoning is not only sloppy, but dangerous.

(a)
If your program is anything but a throwaway or one-off, then there’s no excuse for hacking it out without a detailed requirements document and design/functional spec. I’ve already spotted a few ninja-style programmers in the comments who “just wanna code” and use “but I have unit tests!” as an excuse to prove that they’ve done it correctly even though there’s no supporting documentation. This attitude will burn you in the end. Some people adore unit tests precisely because they are simply code, and believe they’ve discovered a way to avoid the mind-numbing work of documenting things. This is patently wrong. Unit tests aren’t understood by your customers, and they won’t help a maintenance programmer much either (I know from experience).

(b)
Unit tests may prove that a particular class or method works, but contrary to popular belief, it does not -always- prove that it works correctly. As Jonathan Allen pointed out, you are testing the part of your program that is not only least likely to fail, but also the easiest to debug later on and the least expensive to fix. The real problems, the most difficult ones to fix, are the design flaws. Interdependencies between routines and classes. Semantic mistakes. Sequential coupling. Heisenbugs/schroedinbugs. Erratic behaviour under low-memory conditions. Strange errors getting propagated from library (i.e. DB or networking) code due to things like poor networking conditions. Unit tests test help with precisely 0% of these. If design problems account for 80% of the debugging (I’d say that’s a conservative estimate) then even if unit tests halved the time spent on code problems, it’s still only saved you 10%, and that may very well be offset by the time spent creating and maintaining the unit tests.

©
Unit testing is just one tool that can be used in a full QA program. If the people who preach Unit Tests got half as excited about other types of QA (i.e. exploratory testing and code reviews) then I might be more inclined to listen to them, but instead I get the very strong feeling that they consider it an adequate substitute for a full QA program, which it isn’t. There are no best practices. Learn how to do your job, which means knowing precisely how much or how little testing your project requires, and what the best test methods really are. Don’t just assume that unit tests will magically solve all your problems.

I have a real problem with Unit Testing gospel because I’ve seen the above three points in practice. In fact, these assumptions are implicit in many development and consulting shops who claim to use “Agile” methodology. I note that true Agile development can be extremely effective but I am referring here to the vast hordes who use Agile/XP as a buzzword. I’ve actually seen people state that they don’t need documentation and that Unit Tests are just fine for a complete QA program. I’ve seen the result of this thinking in practice. It’s always a disaster. The resulting product never works correctly. The evidence usually seems to indicate that the people who made it don’t even completely understand what it’s supposed to do - they just dove into the code without thinking and did what THEY wanted to do without heed for actual requirements. Ask for documentation and months may go by before you receive anything. Serious errors are fixed slowly. Enhancements, if implemented at all, can take ages.

Bottom line: Unit Tests are often good. Unit Testing Evangelism is always bad.

73

The 12 benefits are mostly correct, but listing the benefits without the drawbacks is like thinking about the benefits of buying a house, changing jobs, or marrying someone without also considering the drawbacks or advantages of not doing so. For the sake of getting us to think in different ways, I am going to list the 12 drawbacks or opposing thought of the combination of unit testing and TDD that are roughly equivalent to these 12 benefits. Warning, some of these are kind of the same, as the initial 12 are mostly based on the same underlying idea.

  1. TDD forces you to prove your code does what you think it should do before you are in a position to know what it should do.
  2. TDD makes it very difficult to refactor, because then you have to refactor all your tests with your code. While techniques for limiting the complexity and length of the code itself are readily available, this is much more difficult to do with unit tests. If you are using TDD and you don’t have way more unit test code than actual code, you are either not very good at thinking of ways your code could be wrong, or you are not very good at simplifying the code under test.
  3. You cannot improve the design of your code without breaking all that unit testing code.
  4. It’s more fun to be fast and flexible and write lots of functionality with a small amount of code and than to be constantly bogged down by unit testing everything.
  5. They trick you into a sense of false progress.
  6. Unit tests provide mostly uninteresting samples of your code that makes people wonder why you even bothered to make that class to begin with. Real world samples of your code already exist in the code base which will provide much more motivation to learn and use them.
  7. It forces you to make a detailed plan before you know what you are trying to achieve.
  8. It forces the cost of bugs to apply to your initial designs that will never be used anyway. It increases the cost of all your non-buggy code to be the same as buggy code.
  9. Code inspections allow for the combining of two minds and the passing on of knowledge and design principals.
  10. It encourages flow, which is a dangerous thing. It prevents you from questioning yourself and encourages you waste hours going down rabbit holes.
  11. TDD makes impractical designs that have to be changed even before they are actually used.
  12. It’s slower than writing code without tests.

To give you some background, I do use unit testing in my current projects. I used to use them a lot more, and I used to do test driven development. But now I basically only use it for very complex algorithms that can be moved into general purpose, reusable, testable pieces of code. Even then, if doing so actually changes it into a bunch of not so complicated algorithms that don’t need testing, then I’m not really sure how useful the unit testing or the abstractions are, unless you see it will be reused a lot, and simplify other code. Even then, it might be better not to have the unit tests. I am all for extremely powerful, reusable, abstractions, but not so much for turning everything into a not so powerful one. The powerful ones I create are so powerful, most developers would never even think of them and skyrocket my productivity and amount of code reuse at times. In the case that, and by the time I am ready to write unit tests, the interface to my class is already designed, but I do write the tests before the implementation, and this does influence the internal design to some extent, for better or for worse. But I do think it is good to write a few different failing tests before doing any implementation. Not that you should write all the tests first, but at least use the class in a few different ways, so you are not intentionally implementing it overly simplified just to make the first tests pass and then have to change it later. Write enough tests first that you can start to think about how the algorithm is going to end up, and try to create an implementation that will make them all pass. If you get bogged down trying to get them to pass, only then should you revert to oversimplifying the algorithm to make at least one test pass and then move forward from there. After all those tests are passing, then you can add a few tests at a time, each group of new tests should illustrate a different way of using the object in a way that does not encourage you to oversimplify any required algorithm adjustments. When you think you have a complete and working implementation, but feel more tests are necessary to find or prevent bugs, only then should you add one test at at time, and run each one right after writing it. I find that rather than to use TDD to create the interface / external design, I like to pretend any class I want magically already exists and is implemented perfectly, and I just use it how I wish I could. Then when the code using it seems complete, I know what interface I actually want for those non-existent classes. This prevents me from wasting time testing and implementing something that will change before it ever really gets used. I am not opposed to the possibility that I could be wrong, but I have read a LOT of things about the advantages of unit testing, and I understand them well. The concepts were so convincing that I immediately adopted them and felt they couldn’t possibly be wrong. Over time, my experience has greatly changed my viewpoints.

1 Like
74

Hi still_dreaming_1. I understand the frustration you might have in thinking of the additional work involved in developing unit tests. Believe me I know considering I constantly get roped into poorly written architectures which either violates or excludes every programming principle. Unit testing didn’t work in my situations since primarily the architecture was tightly coupled to a data source so immutable data sources are impossible if you have spaghetti code which calls the database almost constantly in the business logic. But consider this before you completely write off unit testing. With the growing popularity of NoSQL frameworks, EF and repository patterns (I’m using MongoDB currently in my personal solutino) all of those problems are a cinch to solve. It may also be the architecture that you choose to build on which primarily bars you if SRP and decoupling is not achieved which is the added headache. But for the most part, if you setup an API architecture properly using solid principles and is well designed then you just need to unit test the BL. Also if your like me and choose an N-tier you only need to unit test those (like you mentioned) are complex algorithms or business logic. For the longest time it seemed impossible without some sort of IoC container if you are doing an enterprise service bus but after carefully considering the generics it becomes extremely simple with frameworks like moq ect.

It is also worth mentioning that repetitive testing in web development is most likely to become extinct with frameworks like Selenium. Lastly if you ever ask yourself after you saw the same screen for the n^n time during testing “there has to be an automatic way to regression test” then TDD (unit testing+integration testing+UI) might be for you. If you don’t ask yourself that then by all means continue manual regression tests at your leisure. You may find enjoyment out of it as I know some developers do :smile: