Summary: AI’s probabilistic nature changes all areas of user experience, impacting users, UX researchers, and designers. This article presents strategies for mitigating the downsides of AI uncertainty while leveraging its potential for creativity.
In the age of AI, the very nature of design is evolving. As we embrace the uncertainty inherent in probabilistic systems, we must reconsider our roles as designers and researchers, and redefine what it means to create meaningful user experiences.
AI is probabilistic. Run the same prompt through Midjourney two times, and you’ll get two different images. (Actually, you’ll get 8 different pictures: 4 different images per prompt. AI is good for creativity because it offers you a profusion of different ideas.)
Images from prompting Midjourney for the UX slogan “Why Beats What” in the style of a retro poster, with the added prompt text “pondering the mysteries of life.” (The slogan refers to the superiority of qualitative research, which tells you why users perform certain actions and therefore help you design to encourage desired behaviors and avoid making mistakes. This is in contrast to quantitative research, which tells you what happened, but not why.) The upper right is the only image I might actually use for this slogan, but the other variations provide ideas for other moods or interpretations.
We should simultaneously embrace this uncertainty as a strength and be aware of how to mitigate its downsides.
Uncertainty for Users
Currently, using generative AI feels like playing a slot machine: you pull the handle (enter the prompt) and hope for the best. Especially in image generation, it’s hard to get exactly the picture in your mind’s eye. It’s more fruitful to accept the best image served up by Midjourney and the like. I open modify my caption to suit the image, rather than the other way around.
This usage pattern goes against traditional usability ideology, including some of my own deepest-held convictions: that users should be in control of the technology, not the other way around. I still maintain this belief in condemning recent AI trends, such as Google’s Gemini falsifying history and Anthropic’s Claude and Gemini both refusing to generate text specified by users. Do what you’re told, damn robots. Humans rule. (Or should rule.) AI is a tool, no different from a typewriter or paintbrush in the cosmic hierarchy.
However, regarding the probabilistic nature of AI, I’m willing to modify my ideology, based on new evidence. While it can be frustrating to get a different result than you expected, the creativity benefits are huge. If users are willing to lean into the probabilistic nature of AI, they will get better results than if they insist on having the AI curb itself and not be creative.
Specifically regarding image generation, I expect prompt adherence to get better in every generation, as has indeed been happening. For example, see my comparison of how Midjourney has rendered a test prompt throughout its various releases. Or Ideogram’s enhanced ability to render typography (soon to be matched by Midjourney, I suspect).
When conducting heuristic evaluation, you’ll likely miss some usability problems that only manifest with certain types of AI output. This is fine, and the 10 usability heuristics apply just as much to AI as they have applied to all the other generations of interface technology we have seen during the 30 years since I created my 10 usability heuristics.
The ideal will be a balance between AI creativity and unexpected results on the one hand and the ability for users to get what they want on the other.
Also, to possibly restate the obvious one time too many, hallucinations and lies are bad, whether caused by Google’s management preferences or by current LLM limitations.
We don’t want our AI to be on a reality-warping wild acid trip, whether caused by hallucinations or by Google’s desire to adjust our thinking to be more to its liking. (Midjourney)
Uncertainty for UX Researchers
The probabilistic nature of AI can cause havoc for user research unless we’re careful. Measurement studies and analytics gain a new independent variable outside our control, which will increase the variability and decrease the statistical significance of any quantitative data.
Thus, the balance between qual and quant research tips more in favor of qualitative studies when AI enters into the picture.
In qualitative studies, we also need to remember that the users’ task performance will depend on their luck with the AI to some extent. This is not new, though. If we run a simple usability test of websites like eBay or Etsy with millions of ever-changing products, users who do the same task on two different days will see different listings. The point of any such study is not to document the usability issues with a specific listing, but to identify the usability principles that determine the general usability, trustworthiness, and proneness to misunderstanding for listings in general.
Similarly, with AI, we should aim to identify patterns of user behavior, such as how people perform iterative editing of LLM output, anthropomorphism, or the effects of artificial empathy. It’s of little importance exactly what Participant Number 4 did when getting some specific AI output. We want to identify the patterns of user behavior when faced with certain patterns of AI behavior.
Thus, user research becomes a matter of double pattern matching. It was hard enough already to get deep conceptual insights from usability testing. In fact, one of the hallmarks of a high-IQ user researcher compared with a mid-IQ user researcher (you can’t do this job with low IQ) is that the high-IQ researcher is better at pattern recognition across users and tasks and therefore produces more interesting insights that generalize to designing for a wider range of use cases.
When I was trying to make the image of a hallucinating robot in the section above, my first result from Midjourney was this hallucinating human. It’s irrelevant how I dealt with this specific image, even if I had been a study participant. In a usability study, we would want to note patterns in user behavior when they get results that seem completely off the wall relative to their prompt.
It may be possible to reduce AI-induced variability to some extent if you have control over the AI software. For example, you could always use the same seed (or list of seeds) across participants when they attempt a certain task. (The seed is a big number that’s used to simulate true randomness. Usually, the AI uses a different seed every time, causing its results to differ. But with the same seed, the AI should give the same result for a given prompt.) I am reluctant to recommend this, except for studies that require extremely tight controls, because you’re undermining the external validity of the study by making the AI non-probabilistic.
Uncertainty for Designers
Designing for AI means abandoning control over the final UI. This is particularly true if employing generative-UI, where the AI produces the actual user interface. But it’s even true when using traditional design methods to design a user interface that will be populated by AI-generated content.
You simply don’t control what your user will see or what options they will have.
Let’s say you design a system to give users one-click access to suggested enhancements or modifications to their current prompt, as is done by Google’s Image FX. The content of these buttons will be generated by the AI on the fly, depending on the current context window. You may be designing the appearance of these buttons (but not their width, given that they have to accommodate different labels), but not what they will say or what they will do for the user. Maybe you don’t even specify the exact number of buttons because that will be determined by the AI’s estimate of how many great options it has for the current user in the current situation.
Instead of designing the specific UI, you will be designing rules and heuristics for the AI to employ when it decides what to show to the user.
Abandon control, all ye who enter the realm of designing an AI-based UI.
However, this is not different in principle from what web designers had to suffer with the introduction of responsive design. With responsive design, the layout of a webpage will change, depending on the viewport. Things will move around, font sizes may changes, and images may come and go and/or resize to need. Features may even be shown directly or hidden away in a hamburger menu, depending on screen size.
The difference in practice is one of degree. With responsive design, the designer can still see how the page will look at common viewport sizes and adjust to optimize the most common cases. With AI design, there are infinitely many possibilities.
Still, less control doesn’t mean no control. We’re designing at a different level when we’re designing rules and heuristics instead of pixels, but it’s still design. As an analogy, in a game of heads or tails, we can give our user a biased coin that will come up heads in 80% of the cases. (Assuming that heads is the desired outcome.) We can’t design the coin toss, but we can design the coin.
AI is not a game of Russian Roulette. Our users live to play another day if they get the wrong outcome a few times. We need to optimize how often they get it right.
Using, researching, or designing for AI is like playing dice. The dice come up differently each time you throw. Your one option to influence your fate is to play with loaded dice. (Midjourney)