You’re conflating two issues. You MUST listen to your users, but you need to understand their needs. They DO know what they want. Sometimes what will make them happy doesn’t even make sense, or no matter how wrong they are, sometimes they won’t be satisfied until you give them something they don’t need. There’s just no way around that.
Great Article!
I have stumbled and watched so many others as well who “listen” to their users too much. Designing what the user says they want, rather than really listening, understanding and designing what they really want.
I still struggle from time to time, but life has become so much easier now that I listen correctly.
I think Apple does this well… think about it. The original pre-iPod user wants was something probably close to “I want to listen to my music, let me view any song, any album, sort, shuffle, pause, play and million other things, all with buttons” I can imagine a blackberry style amount of buttons to do that if handled by someone else (and I think I have bought some…), iPods really has only 2 buttons, sure one is more like a touch pad, but it is very very simple, and is easy for people like my parents just to hit “Play”, while I am able to (using iTunes) create complex playlists to let me play graded songs from certain genres, periods of music and artists. It does anything I can think of (for the most part) and yet meets the “do not make me think”.
Don’t listen to your users, but pay attention to how they use your software.
It’s been a while, but Great post! Waiting for stuff like this to come up is why I keep returning here 
Now this article covers just what I meant back when the UnitTest/UsabilityTest article came out a few weeks ago.
As engineers we can get solid numbers about usability and this will almost always trump some designer’s intuition and sense.
This is why I always look at my users using what I make, and why I always use my own stuff for a while before handing it out to others. We can eat out own dog food before having to force my customers to eat it.
What if we listen to what the users say, observe what they do, and carefully merge them.
This post has come at an extremely useful time - so thank you Jeff.
At uni, we’ve just been given basic design specs for a team project we have to work on over the course of the year for an outside client. As this is all of our first real-world applications that we’re going to be developing, it’s really helpful to have some ideas behind us that’ll help us build a more useful system.
Cheers Jeff.
I guess it’s hard coming up with a title, but the titles suck lately, and as others point out, issues are conflated. Lately, this blog feels like it just doesn’t take itself seriously. Pretend we’re all doctors or something, would you discuss medical advances and research in the same tone?
(sarcasm)
I Repeat: Don’t listen to your patients. Patients never know what’s really the problem. They just sit there and complain about a sore bum and headaches and stuff, and never seem to be able to run up their own bloodwork. Man patients suck.
(/sarcasm)
Don’t JUST listen to your users is more apt.
As usual when universals are made and rejected, reality lies in between.
While it’s generally true that users don’t know what they want/need, it’s also often true that some of them have very insightful ideas about how to improve things.
Also, your most vocal users can also be your most vocal supporters. Simply ignoring their complaints/suggestions may hurt your user base.
One more point, the game design domain may not be as closely related to the web design domain. The designers and users are different in significant ways. (Successful) game designers are a lim
As usual when universals are made and rejected, reality lies in between.
While it’s generally true that users don’t know what they want/need, it’s also often true that some of them have very insightful ideas about how to improve things.
Also, your most vocal users can also be your most vocal supporters. Simply ignoring their complaints/suggestions may hurt your user base.
One more point, the game design domain may not be as closely related to the web design domain. The designers and users are different in significant ways. (Successful) game designers are a very limited subset of programmers. They have already proven they can build a popular and usable system – opposed to all the games that fail in development. Web designers and developers are both numerous and have a lower bar to pass. The users tend to be a bit more hard core, perhaps younger, and a bit less “insightful” in their comments than general web users.
That said, your point should is well taken – ethnographic analysis of users (both in the lab and in situ) should always be a major part of application design and revision. “Watching what they do” is a critical of the usability puzzle.
Jeez… so much for editing in a text entry box:
That said, your point should be well taken – ethnographic analysis of users (both in the lab and in situ) should always be a major part of application design and revision. “Watching what they do” is a critical piece of the usability puzzle.
The point is to listen to you user’s needs, but professionally advise their decisions
Don’t believe the steam-resolution data.
Many players lower the resolution before playing Half-Life 2 in order to gain better frame rates. On a notebook with a poorer graphic-card some will even lower to 800x600.
I’ve seen for years no one who uses 1280x960 - most people use today 1280x1024 (or 1280x800 on a notebook), as webstats show. This fact shows that the Steam-Data is totally wrong.
And don’t underestimate marketing: Fulfill the wishes of the high-end, because these are the opinion-leaders, who write in blogs or magazines.
I just like to point out to NurseRatchett that the whole (sarcasm) (/sarcasm) thing was not needed. We got the point. Listen to me - I’m a user.
"I actually play Team Fortress 2, as do some of my friends, and ask any of us why we don’t play medic and you’ll get the same answer without even having to gather detailed statistics."
That may well be true, but in this instance looking at a 5% figure is much quicker in highlighting the problem than collating many responses, most of which won’t be in direct communication with Valve but instead spread around many forums (which, as I understand, they do look at quite a bit). Once you can see a problem, you can ask the direct questions such as “Why aren’t you playing as Medic”. But if you just asked people who gets played the most, whilst I would have said Medic barely gets played I would also have rated spies much higher. Other people would similarly skew figures based on their own experiences, which class annoys them the most, which they think are most valuable etc.
The one item that is missed in all this is that I am keeping existing users happy, but many times I need to make changes to attract new users. Having the 5 happiest users in the world won’t make you a lot of money unless your product can grow to be useful to many more users (or you charge those 5 users a ton of money). I pay attention to both my current users and also try to address the needs of “future” users.
Reading your post made me think of what kind of data we are collecting from our users. We record the errors that happen but nothing about what actually works. We kind of assume everything works if they don’t complain or the application doesn’t record an error.
Instead of an error log maybe have a log that runs once or twice a month that records most used screens, buttons, clicks, keystrokes, etc.
Great post! Thanks.
I’ve found it works best to listen intently to your users, take notes, nod knowingly, promise to do everything the user requests, then go do the software the way you know it needs to be done and later try to convince the user that’s what they asked for…
I think the key is to ask whether you have enough information about the phenomenon you’re studying and whether that information is reliable or not.
Any data can be relevant (user feedback and statistics included), but if you don’t collect the correct data (or enough of it), any action you take will be based on guesswork and personal experience.
Most poor HCI choices are made by people who
- don’t really know what it is they want to know
- asked the wrong questions
- asked too few questions
- have chosen whom to ask inappropriately, or
- think the data they collected has more information than it really does
In the application I’ve been working on for the last three years, we implemented two features that turned out to provide “Big Brother” type usability observation. This is a mainline business application, not a game, so it should show how this is feasible in just about any application.
1.) An internal audit trail. Every change that the user makes to any data in the database is recorded in a database table that records the following data: the date time, the original table that held the data, the column name of the field that was changed, the original value, the new value, the page it was changed from, and the user who made the change.
What this did for us was unintentionally reveal which features the users were actually using. We found that the most used feature in the application turned out to be one that was added late in the development cycle, as an afterthought at the customers’ request. It got a lot of attention in the second release iteration.
It also tells us which users are actually using the system. During testing, it tells us which features actually got tested. For defect resolution, we can see how data was changed, and what data a user was working with when an error occurred.
2.) An internal event log. Any time an exception is thrown, it’s recorded in this log. Any time a user logs in, it’s recorded in this log. Any time a user uses the Log Off button, it’s recorded in this log.
This tells us which users are actually logging into the system, and how many are logging off correctly as opposed to just letting their sessions expire. It also gives us the detailed exception information without having to scrape the Application Event Log in Windows. Because the table includes the ID of the user who was logged in at the time and the date and time it was thrown, we can find relevant information in the Audit Trail at the same time.
Jeff is absolutely right: where errors and usability problems occur, you just can’t rely on users to provide accurate feedback. No one is actively thinking about those details when you ask them about them. Or, they want to present the best possibly face to the developer (who frequently intimidates them) when they’re asked.
Hey Now Jeff,
After reading this it reinforces the importance of UAT (user acceptance testing). When there are testers that look from a users perspective rather than just a quality assurance perspective the produced application is going to be better, even if its something small such as tab order on a login in screen. Nice post.
Coding Horror Fan,
Catto