I Repeat: Do Not Listen to Your Users

blog
43

Yay, Nurse Ratchett - beautifully put. It’s no big secret: Use what data you can get (advisedly); use what user feedback you can get (advisedly).

And “advisedly” applies to both. Eg maybe your feedback does show that they used one particular feature more than others, but, oh I dunno, maybe the other features were too complex for them to understand, so they abandoned trying to use them?

The best way to get user feedback is not just to log them, or listen to them, or even to watch them use it, but to watch them use it having asked them to speak aloud their thoughts as they do. You will learn ten times more about what is wrong with your product this way (and it will stagger you what’s going on in normal users’ heads - that sure ain’t your System Model that they’re working to).

As a designer, not a coder, I have an alternative headline for you:

*** Do Not Listen To Your Techies. ***

:slight_smile: because what techies (and marketers, and…etc) think is a good idea at the UI/feature level usually isn’t, until you have an idea of what is really going on in normal users’ heads. Turn yourself into a half-decent designer by finding out: spend some time doing this spoken aloud user test with some normal users (using your early version, or using competing products, or whatever relevant that you can get hold of). It will change your life - and, hopefully, the lives of your future users.

Of course, few self-respecting techies ever do this - you’ll typically just say to yourself “yeah, sure, but I’m not that bad, I think I understand normal people pretty well, and we do get user feedback, so, no problem, continue as before”. Mean time, normal people are having a truly horrible time with technology, and just hanging in there doing whatever they’ve managed to get to work, too scared to change or add anything in case that breaks their current tenuous usage.

Snappy headlines are the order of the day, so I can’t go with “Do Not Listen To Anyone Who Doesn’t Sit Through Some Spoken-Aloud User Tests In Order To Discover The Real World Of Users”. But since 90% of the people who design and affect the UI/features of tech products are indeed techies, and who don’t do this, let’s go with it again:

*** Do Not Listen To Your Techies. ***

44

Another developer I worked with told me about a story from World War II where the Allies wanted to address the loss rate of planes from bombing missions, which was in excess of 50%. They went to study the planes when they returned to determine where the planes were being most damaged and worked on reinforcing those areas.

However the loss rate did not improve. The problem was that the data they had was from the planes that did make it back. Their damage was in the non critical areas to begin with. The data they needed was for the planes that were lost.

The point about data collecting is good, but make sure you have the right or relevant data is key.

45

I agree 100%. Your examples are gaming based but your points are especially true in my experience with business software.

At my company we have a periodic user/developer meeting to discuss improvements that can be made to our proprietary systems. Without fail about 75% of requested features are never supported by our data. And whenever we are required by the powers that be to add the new features, regardless of what the data suggests, the features are inevitably not used and eventually forgotten about.

Your article is well timed as I have been compiling data the past two days which debunks one such request.

46

One more comment on this. My comment above is presupposing that good, objective analysis is being used when designing and building systems to begin with and not overemphasizing any one source of info such as data or user input.

That being said, one of the best bits of advice I ever received regarding user feedback during analysis is to ask the user why five times. (The actual number of why’s will obviously vary depending on the situation but the basic principal is sound.)

If a user tells you something isn’t working properly or that something would be better if it work another way, be sure to dig down far enough to get to the true root of the issue. And this is best accomplished by asking open ended questions, like why. It gets through all the fluff and does not lead the user in one direction or another.

47

Isn’t that listening to users anyway? In the examples given, developers weren’t literally asking them for feedback or feature requests but they were ‘listening’ anyway?

48

This whole article basically boils down to two (fairly) well-known quotes:
“Actions speak louder than words” - unknown
"One accurate measurement is worth a thousand expert opinions." - Adm. Grace Hopper

49

I personally love the medic class. If your team has a medic and the other doesn’t that usually makes the difference in winning vs. losing.

50

great post! spot on IMHO. my friends and i just launched a website and we’re looking for some user feedback. we’re also interested in seeing how user will really use our site. we’re in a private beta period right now but if you check it out and enter you email address, we’ll get you approved usually within an hour. www.createdebate.com is the address. sorry for posting a plug but, seriously, i can really relate to this blog post. thanks.

51

I’d mostly agree with this. Generally when I go talk to users, it is to educate myself enough to become a user like them. Then I can see what needs doing, what needs streamlining, reorganizing, rearranging, etc.

If you simply listen to them and do whatever they ask, you end up throwing tons of unrealted new features onto whatever their favorite screen or tab happens to be. A user doesn’t really know what the possibilites (and difficulties) are for the software, so you can’t expect them to design things for you. So you shouldn’t let them.

52

Excellent post Jeff. I must admit I’m guilty as charged of believing that user feedback was king of kings. When I did watch a user use one of my applications it was only to mainly see if any bugs would crop their ugly head up in my program. I’m glad that I know better now. With actual DATA to look at AND listening to user feedback I can pick out the more overly critical users from the sane ones. Thanks a million.

53

One of the trickiest things about managing my personal project (Migratr) online has been learning to filter user input. I have a confirmation “Do you really want to do this?” dialog setup for a fringe case that most people hit by accident instead of intentionally, and one user claimed it was “annoying” and I should just get rid of it. That’s the first time I recall ever telling one of my users “no”. Not “I’ll consider that for future release”, just “I’m not going to do that.” He wasn’t a troll or impolite or anything like that- he actually had a fairly decent argument for it. Which, honestly, made it a little more difficult. It’s always easy to ignore a troll.

Currently, my problem is this (and this would be an interesting follow-up post): The applications you mentioned, TF2, Halo etc, are ONLINE games, run through servers maintained by the developers- So all the data they need to analyze to improve their product already passes through a point they can control. How does someone writing a desktop application get data on how their product is being used, without depending on user feedback? It becomes an issue because even if I were to have my app send a small, anonymous packet of information to the server every time it was used, it would quickly be labelled “spyware” and people would get up in arms over it. Even if I had a checkbox, “send anonymous usage statistics, yada yada yada” in an options screen, I’m afraid it would still carry that spyware stigma.

54

You just pointed out the fundamental flaw with many corporate IT organizations. We rely on business users to create requirements for us to execute. They say add a button so I can do this. But if we had dug a little deeper we would have realized that the action they want the button to do, should be done automatically without the additional UI clutter.

At the end of the day most IT professionals don’t do well at root cause analysis of user problems, because we aren’t great communicators. So it’s easier to do what they say, than to get into a meaningful discussion and solve a real issue.

Unless we get data like you suggest. We like data.

55

Take your stinking paws off my software you damn dirty user!!

hehe.

56

In an environment where you have direct contact with the user, the skill lies in figuring out what the real problem s/he is describing. The developer needs to have sufficient understanding of the domain for the user to be able to describe what pain they’re suffering so that the developer can offer possible cures based on their expertise.

If a colleague tells me they need a report, I’m usually off into Five Whys to establish what they are trying to achieve. The solution is seldom what they asked for in the first place and is usually more satisfying to all of us.

Once they’re well-trained, of course, asking for something as specific a report may very mean that their problem actually is that they don’t have the report. Perhaps they need to deliver it to a regulatory body, for example.

57

Using TF2 as the example, part of the statistics analysis also means deciding what, if anything, needs fixing. In order to do that, there has to be an expectation of what distribution is acceptable.

Spy and Sniper are niche classes, so you would expect their usage to be lower than the Soldier, a solid all-purpose character. The question, then, is how far do you let it slip out of balance before you intervene? Is there even a reason to intervene, or were your original expectations too idealistic or failed to consider some other state of equilibrium?

Would the Pyro be next on the list after the Medic gets some love just because he is underused, or is he fine the way he is? Is it the players’ fault that he is underused because they aren’t using him properly? Maybe the maps are too open and don’t take advantage of the Pyro’s love of ambush.

Are the Scout and the Engineer the top two classes because they are too powerful, easier to play, more fun, or more necessary?

There’s a lot more to consider than just raw percentages.

58

Awesome post Jeff, spot on.

59

Back in the dark ages (middle 90’s), we rolled out a new version of our application to a group of production users who had been designated as testers. We had asked them to keep some sort of log about issues they encountered.

After spending a couple of days talking to the users and getting no useful feedback, I implemented something that was radical back then; I added a line to the network Login script that wrote to a text file every time the user logged in. Each day, I would sort through that file and sort it by user. Most users logged in twice a day; once in the morning and once after lunch. There were a couple of our testers who were rebooting a dozen times a day or so. When I asked the testers what was going on, they told me ‘Same as the usual. After the first time it blows up in such-and-such a place, we reboot and things work correctly’. Apparently the application had had that particular problem going back a couple of versions.

Admittedly it was a rather crude way of figuring out what the users were doing, but it certainly helped us find out what was REALLY going on.

60

The title says it all.

nice post!

61

Mike makes a Real Key Point.

The user wants something; however, what they really want, if you ask them enough questions, and what they Iask for/i aren’t always the same thing. A cynic might say they’re Irarely/i the same thing.

Titrat: I was going to mention that*, but I believe it does not prove the data “totally wrong”.

What it proves, more likely, is that the 1280x1024 result got mis-captioned, or at WORST the two categories got combined in the database.

The rest of the data for resolutions “feels” right; at very least it’s plausible.

  • If only because my windows gaming machine has a 1280x1024 monitor.
62

“There’s no need to directly watch users (although it never hurts) when you have detailed logs showing what they actually did.”

Logs only show what people have done… watching the users directly gives you an idea of how quickly they made decisions (bad/confusing UI), and you get an idea what they were trying to archive at the time, compared to what they actually did.

For example… logs which show the hit count for a websites site map, don’t show you if the user went there by mistake, or if they chose that route because the navigation bar was not understood.