Summary: Advising graduate students, Jakob Nielsen champions a blend of heuristic evaluation and usability testing and urges young UXers to avoid being the sole UX talent in startups. Stakeholder observation is his secret sauce to build credibility.
In September, I met (through Zoom) with the HCI Masters students at CMU. Carnegie Mellon University has one of the world’s best UX graduate programs and brilliant students. (I highly recommend hiring these high-IQ UX talents in a few months when they join the job market.) I enjoyed the meeting tremendously and was happy to give some advice to the next generation of UX leadership.
Jakob in a mentoring session with the 2024 cohort of MHCI students at CMU. I spoke from my home in California, which was nice and warm.
One student asked me how new graduates (as these students will soon become) can motivate and encourage their teams and higher-ups to implement and allocate money and resources towards usability testing and heuristic evaluation despite “being at such an early stage in our careers where we might not be taken seriously.”
Here’s my answer (lightly edited):
There are two different aspects of what you're asking. First, there is actual user testing — a usability study where we watch a user or several users. Second, there's heuristic evaluation, where you don't watch users but use your expertise to evaluate the user interface. As the originator of heuristic evaluation, I obviously believe in that method. However, I don't believe heuristic evaluation is the only way to improve user interfaces, and it’s also less good at building credibility for UX newbies.
I recommend combining heuristic evaluation with usability testing and alternating the two. Don't waste users on finding glaring deficiencies in the design you can find in a review while using the 10 heuristics.
Why Heuristic Evaluation Works
The arguments for the two methods are slightly different. To take the heuristic evaluation first, the argument for heuristic evaluation is that we know things about UX. Unfortunately, some people don't believe in this. Some people say that the only thing you can do is just run an A/B test and see what gets the most clicks. And that’s the only valid data. I disagree with this perspective. Now that I have done UX for 40 years, my experience is that we actually do know certain things about UX.
Some designs are better than other designs. Possibly, we don’t know this to 100 percent perfect accuracy, but we know it with maybe 90% accuracy, which is good enough. We are not flipping a coin or going with the designer's first guess or anything of that nature. Instead, we can apply these extremely well-known principles that have been proven for decades. The 10 heuristics are not some random ideas I got yesterday. These are very, very well-established insights like visibility of system status. The heuristics are exceptionally well documented and have been repeatedly proven to work. So, we actually do know things.
If we did not know anything about UX, why are you students wasting a year at university? If we don’t know what makes for good UX, then it’s not worth attending classes.
When presenting a heuristic evaluation to a corporate audience, the heuristics lend you some credibility because you are not just presenting your personal opinions (which would be backed by very limited experience, to say the least, if this is your first project). You have the backing of decades of usability research. Still, presenting findings from user testing is the better way to have credibility with stakeholders. (Dall-E.)
Of course, new things always come up where we don't know the answer to what’s good or bad. But one of the most important realizations about the entire usability field is that there are a lot of repeated user behaviors. There are immense gaps between the human mind and the way computers work. These persistent insights lead us to conclude that a certain design will usually work better than another design.
40 years ago, when UX was a new field, and I started working, we didn’t know as much. I wasn’t the first UX pioneer. The field started back around World War 2. But UX was still new when I started. In 1983, it might have been more reasonable to say, “We don't really know anything because we haven't done that much research yet.” But by now, we have. And so now we can really say that we know the established principles. We can apply those principles to evaluate user interfaces.
The beauty is that my heuristics are not your opinion. That’s why the heuristics are essential, because if you just say, “I don't like this design,” why should that be any better than some other person saying, “But I do like it.”
With heuristic evaluation, you can refer to an established principle for each point. You can point out why this principle applies to this design situation. The heuristic is why I’m recommending what I am. When you are young, you have less credibility, just to be honest, in making these theoretical arguments. But it’s still a solid argument to make, and I’ll have your back in the form of the heuristics.
User Testing Lends Credibility to Young UXers
It’s different on the empirical side of things: testing with real users. It doesn’t matter how young you are, how green you are, how few products you've worked on. All that matters is that you use a reasonably good study methodology. But I only say “reasonably” good because you’ll get insights even without using as perfect methodology as people who have run studies for 10 years.
The real difference is between no study and some observation of real use. The difference between zero and something is much more important than whether the observation is conducted with different degrees of skill and rigor. If you do terrible research that’s 100% biased, you might get some wrong results, but it has to be really bad for a usability study to be worthless. Something like pointing to the button on the screen and saying to the user, “Don't you want to click that? Isn’t it a good-looking button?” If you do things like that, then your study falls apart.
Presenting findings from user testing? You have instant credibility, no matter how young you are. The test users provide credibility because your presentation is the only time the stakeholders see their customers. (Do show video clips from your study, if at all possible: let the customers talk for you.) Image by Dall-E.
If you minimally follow the methodology guidelines for how to do user testing, then you're going to get a decent number of good results. You might get more good results 10 years from now when you have more experience, but you're already going to get good results now, which is the critical point.
Get some of your stakeholders to observe the research to build your credibility. Ideally, they would be with you, and it could be literally with you in a lab or a conference room. Or it could be virtually in a Zoom call. Just make sure they go on mute so they don't bother the test participant. Watching a real user is critical, whether your observer is a developer, a manager, or a marketing executive.
Stakeholder observers will inevitably see two or three of their customers persistently clicking the wrong thing, persistently verbalizing the completely wrong interpretation of what the system is supposed to be doing, or they’ll witness another of those observations we typically get in early user testing. I should also say that there’s a famous cliche about low-hanging fruit, which is super true. The first time anything goes through usability testing, there will be plenty of low-hanging fruit. There will be glaring problems.
Low-hanging fruit: there will be plenty the first time you run a usability test on a design that hasn’t seen much UX love in the past. Identifying these fruits is an excellent way for a young UX professional to build credibility with the team and management. (Image by Leonardo.)
Benefits of Reporting to a Professional UX Manager
On the other hand, you may have a very polished user interface that has been through years of thorough UX work. In that case, if you are assigned to fix one little thing, then maybe the results will not be so obvious. Then, getting stakeholders to watch your study may not be quite as convincing. On the other hand, in that case, you will be working in a highly mature organization because that's the only place you will find a polished design that has been through years of great work already. In that case, you can rely on having a good UX manager who can defend you, and it will be his or her problem to convince the other managers. It’s not going to be your problem.
So, your credibility is more of an issue if you’re on your own, which I don't recommend for new graduates. Career-wise, it’s very tempting for people like you to go out and be the so-called Unicorn in a startup. Maybe if you get some stock options, they’ll be worth $10 million, but most likely, they'll be worth 0.
If you’re the only UX person in a company, you risk what I call “being raised by wolves.” You learn all bad habits and wrong methods. Whereas if you join a company with a professional UX manager and professional colleagues who have already been there for some years, then you can soak up a lot of practical experience from them. Maybe in your second job, you can become the one UX person who’s a hero and makes a startup work. But I recommend against solo UX work for a first job.
Raised by wolves: a grave risk when a young UX professional takes a job as the sole UX person in a company. (Wolves’ den by Leonardo.)