Closed vs. Open Questions in User Research: AI Makes Free-Form Answers Cheap to Count
Summary: Closed-ended questions measure only within the categories you supply, whereas open questions let users reveal what those categories missed. AI now conducts interviews, codes every verbatim, and counts how many respondents raised each theme, so you can finally afford to analyze every open answer. Shift your question mix toward open questions while keeping a fixed closed core for benchmarks, and never stop watching what users do since every answer is self-reported data.
Watch my 24-minute training video about Closed vs. Open Questions (YouTube).
Alice and Zimo walk through both kinds of questions in a 32-page comic book drawn in Fully Painted Gouache Album style. Almost as long as my 35-page History of AI comic. (GPT Image 2.5 Sunburst)
Two Doors: Mail Slots and Open Arches
Every question you put to a user is a door. Some doors are a row of numbered mail slots, and the respondent can only push a note through one of the openings you cut in advance. Other doors are open arches, and the respondent strolls through, arms full of whatever he or she wants to tell you.
Definition: A closed-ended question supplies the possible answers in advance (yes or no, multiple choice, a rating scale, a ranking), and the respondent picks from the list. The researcher has already decided what can be said, and in what words.
Definition: An open question, AKA an open-ended question, supplies only the question. The respondent answers in his or her own words, at any length, about anything the question brings to mind.
Example: “How satisfied are you with checkout, from 1 to 5?” is closed. “What happened when you tried to check out?” is open. The first tells you that 23% of customers picked a 2. The second reveals that the coupon field rejects codes typed in lowercase, a problem you hadn’t thought to ask about.
Closed = Count, Open = Discover
What’s the central difference? When the categories are fixed. Closed questions impose them before users answer. Open questions let you develop them afterward, once the answers have shown you what the categories should be. AI makes that after-the-fact classification cheap, but whoever writes the categories still decides what counts.
Closed-ended questions turn 1,000 respondents into one chart, cost each respondent a few seconds, and give you the same scale next year for tracking trends. But their blind spot is built in: you only learn about what you thought to ask. An ignored “Other” box swallows the surprises, while a forced answer manufactures opinions where none existed.
Open questions uncover problems nobody on the team imagined. One customer writing “I gave up and called my nephew” moves stakeholders more than any bar chart. For decades, the expense of hand-coding thousands of answers turned the write-in box into a graveyard: respondents took the trouble to type, but nobody ever read what they wrote.
When to Use Which: Open Early, Closed Late
Start open. In discovery, a premature list of options fences the study inside the team’s current understanding, and the problems you most need to find lurk outside that fence. Once users have revealed the relevant categories, switch to closed questions to measure how common each one is and to track changes after a redesign. But keep an open question in later rounds, too. A benchmark will faithfully track yesterday’s problem while tomorrow’s problem walks in unannounced.
The door you build decides what users can bring you. Build both kinds. (GPT Image 2.5)
Within a single study, pair them. Ask the score, then ask why. A rating of 2 out of 5 signals dissatisfaction, and the free-text follow-up tells you where to dig. Skip the second question, and you’re left with a dashboard of unexplained numbers. That’s the sad fate of most survey programs.
Both kinds of questions collect self-reports, whether a human or an AI asks them. The oldest rule in usability still applies: watch what users do, not what they say. (My usability slogan number 8.) In the comic book, the man who “always reads the instructions” has the manual wedged under a wobbly table leg.
Questions capture attitudes, memories, and intentions. Only observation reveals behavior, whether through analytics or usability testing. When self-reported answers conflict with what you observe, believe your eyes.
7 Guidelines for Closed-Ended Questions in Surveys
Make options mutually exclusive and collectively exhaustive. “1–5 times” and “5–10 times” overlap at 5, and a customer who did it 11 times has nowhere to sit.
Balance the scale. Two positive options need two negative ones, with a neutral midpoint, or you’ve built a leading question out of arithmetic.
Offer an honest exit. “Not applicable” and “Don’t know” cost you nothing and save you from opinions invented on the spot.
Ask one thing per question. “Was it fast and easy?” is two questions harnessed to one cart. Split them.
Never lead, and guard against anchoring. “How much did you love it?” has already chosen the answer. “Would you pay $20?” before an open pricing question plants a reference price, so the next answer stays tethered to that anchor.
Ask about recent experience. “Would you use this feature?” invites cheap promises. Ask what the respondent did in the past 7 days and what happened.
Keep it short and pilot it. Every extra question sheds respondents. Run the questionnaire past 5 people before you field it, and watch where they hesitate.
4 Guidelines for Open Questions in Surveys
Ask about one specific moment. “Any comments?” offers respondents no handle on their memory. “What was the most frustrating moment of your last purchase?” hands them a concrete event to recall.
Make the box fit the task. Provide enough room for respondents to see and edit a useful answer. A cramped one-line field whispers, “Keep it short.”
Keep them optional and spend respondents’ effort wisely. Put broad write-ins near the end, but place a diagnostic follow-up beside the question it explains.
Always finish with a catch-all. “Is there anything else you want to tell us?” leaves room for concerns your research agenda missed.
Interviews: Open Questions Do the Work
In a user interview, the balance flips. Open questions carry the session, and closed questions play a bit part. “Walk me through the last time you paid an invoice” starts a story. “Was it hard?” ends one in a single word.
Three tools separate a good interviewer from a form with a pulse. Silence gives participants time to recall what a hurried answer leaves out. Probes open the door behind an answer: “What happened next?” and “Can you show me?” And neutral wording leaves the account in the participant’s hands, whereas “Was it confusing?” has already written the script.
Closed questions still earn their keep in interviews when you need to confirm what you heard. “So you switched apps in March, before the price rise?” pins down a fact and a date. Then open the door again and let the story run.
The same two doors appear wherever you ask users anything: one score plus one “why” after a usability task, or one question (two at most) in a website intercept. Support tickets, app-store reviews, and diary studies pile up still more first-person accounts that deserve the same analysis.
When AI Asks the Questions
The interviewer no longer has to be human. AI can interview 5,000 customers tonight in 40 languages and stick to the discussion guide from the first session to the last. It won’t even sigh when the 40th participant describes the same login problem in the same words as the 39 who came before. That changes my advice in two ways.
The AI interviewer sounds as fresh for participant 5,000 as for participant 1, with no boredom seeping into its voice. (Muse Image)
First, the static questionnaire becomes a conversation. AI can improvise follow-ups from each participant’s own words. Traditional survey logic can only march down branches that somebody scripted in advance, even when a respondent drops an intriguing clue that begs for a follow-up. Keep a fixed core of closed questions for benchmarks, with consistent wording, asked before any adaptive probing. Otherwise, a different conversation can prime a different score. Identical wording after different conversations only looks comparable.
Second, the AI needs interviewer training. He Zhang and colleagues at Penn State tested InterviewBot, a wrapper around a June 2025 version of GPT-4o (now obsolete), with 15 participants. The bot probed too little and gave participants leading feedback. Only 4.9% of its turns were probes that dug deeper. It swallowed thin answers whole and praised responses in ways that signaled approval. Praise is a leading question in disguise. My rules for a machine interviewer:
Ban evaluation. No praise, no “great point,” no reaction beyond acknowledgment.
Probe for missing evidence. When an answer lacks a concrete episode, ask for one. Once the account is clear, move on.
Permit “unclear.” Let the bot admit when it can’t make sense of an answer. Otherwise, it will invent a meaning.
Disclose and release. Tell participants they’re talking to an AI, let them skip, let them stop, and keep a human on call for sensitive topics.
Pilot with 5 humans and read the transcripts yourself. You’d pilot a questionnaire. Pilot the interviewer, too.
When AI Reads the Answers
Leonardo Bergmann and co-authors from the University of Vienna and other European universities compared 5 human coders with GPT-5.4 on inductive content analysis of 903 open-ended answers from a European PhD student survey. (I discussed the study in my September 14 newsletter.) Inductive coding develops categories from the raw text, then groups them into themes. Human–AI agreement reached 0.61 for codes and 0.54 for themes on the Adjusted Rand Index, a chance-corrected score for how closely two groupings agree (1.0 would mean identical groupings, and 0 is what chance alone would produce). That’s good enough to give AI a seat on the coding team, as long as a human reviews its work.
What happens when reading becomes free? The open question becomes a quantitative method. AI makes it practical to code every answer and count respondents under each theme. Verbatimetrics, the counting of coded verbatims across the full dataset, keeps the words attached to the numbers. A result such as “37% of respondents mentioned the coupon field” comes alive when the answers behind it are one click away. Count each respondent once per theme, however many times he or she repeats the complaint.
But mention frequency measures what people chose to report. A customer may encounter a problem without recalling it, prioritizing it, or bothering to write it down. So the silent 63% haven’t declared the coupon field trouble-free. To estimate how many customers hit the failure, ask a targeted question of everyone who tried to use a coupon, or check the analytics for failed coupon attempts. Open answers reveal what stuck in people’s minds. A valid prevalence estimate requires giving every affected customer the same chance to report the same event.
Nor should frequency dictate priority. A frequent irritation may cost seconds, while a rarely mentioned failure blocks a sale. Use the coded counts to locate patterns, then check what each problem costs users and which tasks it breaks before you set the roadmap. One payoff of coding every answer is catching the rare but costly exception that a summary of the biggest themes would bury.
Two findings from the coding study explain why you must keep the original words within reach. First, the AI sometimes forced unclear answers into confident categories, laundering noise into clean-looking data. Let it mark an answer “unclear,” and eyeball the short or ambiguous responses yourself.
Second, agreement was lower for broad themes than for specific codes. Keep humans on theme duty, because that’s where the model lumps distinct answers under umbrella themes so roomy that a dozen different problems fit underneath. File coupon rejection and payment failure under “checkout friction,” and you get a tidy report that hides two different repairs. (Also run a frontier model at maximum reasoning effort. The study used a model that’s already behind.)
The codebook becomes part of the measuring instrument. Freeze its definitions when comparing waves of research. If you change the model, prompt, or categories, recode the earlier answers with the new setup before interpreting a trend. Otherwise, an apparent rise in complaints may only mean that the machine now draws a wider boundary around “checkout friction.” Keeping the raw answers gives open questions an advantage no closed question can match: next year, you can reanalyze last year’s answers with updated categories, while closed answers stay frozen in whatever options you happened to offer at the time.
AI removes much of the analyst’s labor, but respondents still pay with their attention. So spend the savings on better questions and on investigating what the answers reveal. Never field an open question you don’t intend to analyze, and don’t let cheap coding excuse an exhausting questionnaire. (To bastardize the title of a famous AI paper, user attention is all you need for representative responses, and customers bestow precious little of it on your research. Don’t squander it.)
Every question spends some of the user’s attention, whether it’s open or closed. Don’t squander it by asking questions whose answers won’t change your design. (GPT Image 2)
AI Can Draft Your Questions but Can’t Be Your Customer
Agentic AI can draft the questionnaire, help analyze a pilot with real participants, flag muddled questions, and code the results. Put a human gate before fielding, and archive the exact version participants answered. Once collection starts, silently “improving” a question is like switching rulers halfway through a measurement. The machine will ask anything you let it. Deciding what’s worth asking is still your job.
AI also offers to be the respondent, and here I’ll be blunt: don’t let AI answer your questions in place of customers. Yuanzi Li and co-authors compared 18 language models with human answers from large social surveys and found that aligned models favored kinder, more socially approved answers, a benevolence bias that grew with model size. A synthetic respondent can describe checkout frustration in convincing detail without ever having touched your checkout. Running 10,000 simulations instead of 10 adds no customer evidence. It only dresses up the model’s assumptions in extra decimal places, staging precision theater. Let synthetic respondents help you brainstorm questions. The verdict belongs to real customers.
Conclusion: Two Doors, Use Both
Count with closed questions. Discover with open ones, and let AI code what you find. Keep benchmarks stable, distinguish mentions from prevalence, and trace the numbers back to the experiences behind them. AI makes reading every answer affordable. Your job is to decide what those answers justify changing.
AI is exhuming the write-in graveyard, one buried answer at a time. What matters now is whether the recovered answers change the design. Open both doors, listen to what comes through, and then fix what you hear.
The three contributors to user research each play their own part. Don’t let AI audition for the customer’s role. (GPT Image 2.5)
Watch my 24-minute training video about Closed vs. Open Questions (YouTube).
