Pop quiz, hotshot. How do you know if your application works? Sure, maybe your app compiles. Maybe it passes all the unit tests. Maybe it ran the QA gauntlet successfully. Maybe it was successfully deployed to the production server, or packaged into an installer. Maybe your beta testers even signed off on it.
The makers of Camtasia, TechSmith, have another product called UserVue. I have found it extremely simple to use, and with a monthly rate it is affordable for when I need it.
Greate article, very clear.
Great article, but you forgot to mention one part of the studies I found extremely enligthening (I don’t remember whether it was Krug’s or Nielsen’s insight, I think it was Krug’s cause I remember the graphics but I don’t have the book on me right now): doing two tests with 3 users each finds out more bugs than doing one test with 6 users, or 8.
Because some bugs or issues may shadow or completely hide other bugs or issues, making users unable to run in them or find them, testing the same application version on 6 users will never be able to find some of the bugs/issues, while 3 users will find most of the bugs the 6 users would find (let’s say 75-80% if we use Nielsen’s graph), but correcting these issues and then testing again with 3 users will end finding out more bugs/issues than the 6 users test would’ve uncovered at a marginally higher cost.
We have an app with a very specific business purpose. Somebody off the street would have no idea what we’re talking about. Is there still value in testing release n+1 with experienced users? Or just testing the current release with them?
Steve Krug’s book is invaluable. I have owned it for 5+ years and still refer to it when it is time to conduct usability tests. It doesn’t hurt to subscribe to Nielsen’s Alertbox either.
Usability testing is, of course, very useful. But these “usability test is easy and cheap” articles seem to me to implicitly be about testing software aimed at the general population. That is, a program where just about anyone is supposed to be able to just sit down and do something. This makes it fairly easy to get user testers.
But what if your product is a niche product in a specific industry where it is assumed that your users will already understand all the basic concepts of that industry and will already understand the jargon of the industry and let’s make it even harder and assume that people generally receive training in your product before using it.
For example, something like Photoshop. A program where you don’t really expect the average computer user to be able to just sit down and show you whether your changes to the “inverse gamma dithering defractor” interface make the program easier or harder to use.
Sure, your staff understands the program. But trying to get external usability testers can become more and more expensive and complicated the more specialized your product is. And I get a little frustrated because these “usability testing can be easy and cheap” articles tend to concentrate on the best case scenerio.
Can you explain how Nielsen has managed to define what 100% of usability errors are, and if you only have to use 15 users to attain 100% usability.
I’m always disappointed at how many people involved in a project avoid usability testing as it interferes with their ability to advance their own agenda. I can’t count the number of times a BA or PM or whoever has approached my cubicle for design advice (I have a design/usability background) and we have had this kind of conversation:
[translation: ‘tell me how great you think my idea is’]
Me: “Well, the command buttons and form fields would be more effective if they were grouped logically. I would recommend arranging them like this (draws sketch) and running it by a couple users in a quick usability test to see how it works.”
[translation: 'asshole. who asked you anyway, mister “expert?” i’m gonna do it my way ‘cos I worked hard to make this page and i don’t wanna see you mess with my design.’]
People – /especially/ non-designers – invest so much of their own ego into their designs, that it becomes a major barrier to even getting any usability testing done, because they’re afraid it’ll punch holes in what they did. Which of course it shouldn’t be viewed as ‘punching holes’ in the first place, but there you go.
I work on a small web development team of four people and time is definitely scarce so the step of usability testing normally gets skipped or quickly bypassed during the development life cycle so that the next project can come down the pipe but we have had several projects that have had some rough launches because of this.
Thanks for all the great tips to making the process easier so that testing doesn’t have to get overlooked and bypassed in order to “save” time.
Great artikel and great site. I come here often, but have noticed one particular behaviour on your site. I am not sure it is intended, but since you’re on the subject of usability, I’ll mention it. I (and maybe I am not alone in this) have a habbit when reading online, to select 3 or 4 lines of text with the mouse (arround where I am reading at that moment). When I do this on codinghorror however, The selection is always at a different line than the selection I make with the mouse. This is strange and sometimes trying ro select things causes a jump to the top of the page.
Great entry. 
stefve - I’ve noticed a similar behaviour in IE on this and other sites. It works fine in Firefox.
I’ll second your recommendation of Krug’s book. It is an invaluable source… and it doesn’t matter if you write web pages (which is kinda who Krug is targeting) or desktop apps.
Makes me think of those Media Player skins, or the ugly skinned disasters both Roxio (and worse) Ahead put on their coaster-toaster software. Even IE7 got a ridiculous anti-user UI facelift.
Unfortunately, Usability is not something we can automate, otherwise we could start a Usability Driven Design movement to supplant Test Driven Design as a coding methodology.
Joking aside, from a UI perspective, UDD makes absolute sense.
As i’m in the middle of a website usability report for a uni module i’ll chime in … “think aloud” has it’s problems: who’s used to talking about their cognative processes whilst undertaking a given task and being monitored? As usual, the best results come from a range of techniques.
If I remember well, the referenced chapter is of the first edition of the book. In the second one he removed that chapters that explained how to do a usability test and put them on the web. I think they are the best resource about how to start doing it.
Can you explain how Nielsen has managed to define what 100% of usability errors are
It helps if you read the Neilsen articles I linked in the body of the post:
Why You Only Need to Test With 5 Users
http://www.useit.com/alertbox/20000319.html
Guerrilla HCI: Using Discount Usability Engineering to Penetrate the Intimidation Barrier
http://www.useit.com/papers/guerrilla_hci.html
Different perspective, but same basic themes in the Krug book. Usability testing is only as difficult as you make it.
“the results are often eye-opening” - So true! Yeah, programmers cannot comprehend the stupidity of users.