Share article
Try picturing two versions of the same customer. In the first scenario, they give you 6 out of 10 on a satisfaction survey. In the second, they give you the same 6, but add: “The product’s fine, but I waited three weeks for my order and nobody told me what was going on.” Same number. Completely different information.
This is something that’s fascinated me about customer experience for years: numbers tell us what happened, but almost never why. And “why” is exactly what we, as CX professionals, need to work with, because a score on its own can’t be fixed. Only the specific problem behind it can be.
Open-ended questions are the tool for getting at that “why”. The trouble is, in practice they’re often used badly. Either we phrase them so vaguely that people don’t know how to answer, or we cram so many into a survey that half our respondents give up before the end. Then we’re left with hundreds or thousands of free-text answers that nobody has time to read. Let’s take a more systematic look at this.
How to ask so the answer is actually worth something
When I first came across the research by Jon Krosnick, a respected scholar in survey methodology, I was almost amused by how many different things get lumped under the label “open-ended question”. Krosnick’s work, published in the Annual Review of Psychology, shows that different types of open-ended questions serve completely different purposes and if you don’t grasp that distinction, you end up with answers you have no idea what to do with.
There are exploratory questions, ones that open up a topic you don’t yet know much about: “What challenges do you most often run into with us?” This kind of question makes sense early in research, while you’re still building a hypothesis. Then there are questions aimed at a specific experience: “What exactly happened when you tried to return an item?” This works best right after the interaction, while the memory is still fresh.
A practical rule I always stick to myself: one question, one topic. It sounds obvious, but I’ve seen dozens of surveys that ask about “the product, delivery and communication” all in a single open-ended question. The respondent writes one sentence about whatever comes to mind first and stays silent on the rest. Not because they don’t care but because a combined question doesn’t give a clear signal of what you’re actually asking about.
Another thing we underestimate: specificity. “What could we improve?” sounds welcoming, but it’s so broad the respondent doesn’t know where to start. It works much better to narrow it down to the context they’ve just experienced: “What would have made today’s delivery easier?” The respondent has something concrete to think about, and you get an answer you can actually act on.
And then there’s something I’ll admit I got wrong myself for a long time: phrasing questions in a way that already hints at the answer. “Didn’t you find the delivery slow?” In a question like that, the respondent senses what you want to hear, and often adjusts their answer accordingly, even if they’d otherwise have said something different. A neutral phrasing “How would you describe the delivery speed?” leaves room for the respondent to answer in their own words, rather than confirm your suspicion.
How many open-ended questions is the right number
This figure genuinely surprised me while researching this piece. Pew Research Center, one of the most respected research institutions in the world, found in its analysis that open-ended questions have an average non-response rate of 18%, compared with just 1–2% for closed questions. In other words, nearly a fifth of people simply skip an open-ended question. That’s no accident it’s real mental and physical effort. Answering a closed question means clicking an option. Answering an open-ended one means putting a thought into your own words and writing it out, which, according to the same source, takes proportionally longer and demands more cognitive effort, especially on mobile.
Pew also found something else that should concern us in CX on ethical grounds: non-response rates vary by education, age and ethnicity, with younger people, people with less education, and minority groups answering open-ended questions less often and more briefly. In other words, if you rely mainly on free-text responses, you risk having your “voice of the customer” actually represent only the segment of customers who have the time, motivation and language confidence to write longer sentences.
So what’s the right number? An analysis by the platform Caplena, drawing on more than a hundred thousand research projects, found that response rates stay reasonably healthy up to around five or six open-ended questions. Beyond that threshold, both the non-response rate and the share of blank or nonsensical answers rise sharply. Qualtrics, in its methodology guide, recommends an even more conservative approach: unless your respondents are exceptionally motivated, stick to five open-ended questions or fewer, and space them out through the survey rather than placing them back to back.
For a typical transactional survey the kind sent after a purchase or a customer support contact that means, in practice, one or at most two open-ended questions. One tied to the score itself (“What’s the main reason for your rating?”), and possibly one targeted at a specific area you’re currently investigating. Anything more is a luxury you can only afford in in-depth, voluntary research with highly motivated respondents not in everyday feedback collection.
What to do with all that text: from reading to spotting patterns
This is where I most often see CX teams get stuck. They collect hundreds of comments, one person is tasked with “going through them”, and after two hours of reading, what emerges is an impression rather than an analysis. Andrew Chen, who has spent years working on growth metrics, put it well: the most actionable part of NPS (Net Promoter Score, a loyalty metric based on how likely a customer is to recommend a company) analysis isn’t the number itself, but the categorisation of open-ended comments from both promoters and detractors.
In practice, categorisation means assigning each comment to one or more themes from a predefined but evolving list: delivery, price, product quality, support communication, and so on. At lower volumes say, up to a few hundred comments a month one person with a spreadsheet and some discipline can manage this. Once volume grows into the thousands, manual categorisation becomes unsustainable, and that’s where text analytics comes in: software processing of free text using NLP (Natural Language Processing).
It’s worth being clear about what text analytics actually does, because there’s a bit of magical thinking around it. These tools typically do two things: identify the topic a respondent is writing about, and estimate sentiment whether they’re speaking about it positively, negatively or neutrally. It doesn’t work like mind-reading; it works through statistical pattern recognition, which is why it performs best on large volumes of text and worst on subtle nuance, irony, or mixed feelings within a single sentence.
One real-world example that really caught my attention while researching this piece: a media company with roughly a million monthly users was receiving more than 10,000 responses to a short satisfaction survey and couldn’t process them faster than once a month. After introducing automatic categorisation and sentiment analysis, along with automatic alerts for negative comments, they were able to identify and address problems almost immediately. Within three months, they raised their score by eleven points. That’s not magic from the technology itself it’s the result of feedback finally reaching the people who could actually do something about it, while it was still fresh.
Which brings us to a point I think matters more than which software you pick: closing the loop. Categorisation and sentiment tell you what’s happening at an aggregate level. But an open-ended question is still, first and foremost, a message from a specific person and if that person clearly names a problem, it deserves to reach the specific person who can fix it, not just end up in a quarterly slide for management. CX methodology publications describe this as the difference between a programme that “has a nice dashboard” and one that actually changes the customer experience.
Strong and weak questions, side by side
I think this comes across best through examples.
Weak: “Any comments?” This question offers no anchor the respondent doesn’t know what to comment on, and the most common answer will be “no” or a blank field.
Strong: “What would have made today’s order easier?” It’s specific, ties to a fresh experience, and signals that the answer will lead to a real change.
Weak: “Were you satisfied with our support and the product?” This combines two different topics into one question, so the respondent answers only one of them or neither.
Strong: “What specifically led you to rate today’s support interaction the way you did?” It targets a single topic, references the rating the respondent just gave, and gives them a clear frame to work from.
Weak: “Don’t you think delivery could be faster?” This suggests the answer, and the respondent senses what’s expected of them.
Strong: “How would you describe the delivery speed for this order?” A neutral phrasing leaves room for any answer, positive or negative.
Takeaways
An open-ended question isn’t a nice-to-have alongside the score it’s a tool in its own right, with its own rules. Phrase it specifically, and around a single topic. Don’t overdo the number of them; for routine feedback, one or two is enough. And above all, have a plan for what you’ll do with the answers before you send the survey. Text nobody reads is really just another form of a number nobody explains.
I’ll admit I’m still surprised by how few companies take the link between question design and data quality seriously. Knowing how to ask is a skill, just like knowing how to listen. And both can be trained.
