Share article
A few years ago, I spent an entire weekend reading customer comments about a banking app. There were just over a thousand of them, and I told myself I could handle them manually, because surely I understand people better than any algorithm. After two hundred, my head was spinning. I’d mixed up two similar themes, and at one point I genuinely couldn’t tell whether the comment “Brilliant, waiting three days for a reply” was meant sincerely or sarcastically. That’s when it hit me: the problem wasn’t that I didn’t understand people. The problem was volume. There was so much of it that I couldn’t read it all properly, let alone keep it all in my head. And that bothers me, because behind every one of those comments is someone who took the time to write down what was bothering them, and the company simply doesn’t have the capacity to respond properly.
This is exactly where text and sentiment analysis come in. It’s a set of techniques that can process free-form text automatically, assign it to themes and estimate whether a response is broadly positive, negative or neutral. It sounds like a dry definition, but it’s one of the most interesting areas of CX (customer experience), because open-ended responses hold the one thing that scale-based questions like NPS (Net Promoter Score, a measure of how likely customers are to recommend a company) can never tell you: why. To understand the limits of what a machine can do, I first need to explain how it actually reads.
How a machine reads text
The basic mechanism is simpler than you might expect. The model first breaks the text down into smaller units, such as words or phrases, and then compares them with what it has learnt from a huge volume of other texts. From this, it can work out two things. First, what the text is about. This is handled by what’s known as topic modelling, which groups similar responses into themes such as “delivery”, “price” or “staff behaviour”. Second, what mood the text conveys, which is where sentiment analysis comes in.
To make this less abstract, let’s try it on a single sentence. A customer writes: “The app crashed on me again today, that’s the third time this month. Really annoying.” A well-configured system will pull out the theme (app stability), the sentiment (negative) and possibly even an urgency signal, because “the third time” points to a recurring problem rather than a one-off glitch. This is the moment when a thousand similar sentences turn into a clear picture of which themes come up most often and most strongly in your data, without you having to count anything by hand.
I’d like to pause on something that confused me for a long time. Sentiment analysis doesn’t work like a dictionary where “great” is positive, “terrible” is negative and that’s that. More modern models, particularly those built on large language models, look at the context of the whole sentence. They can tell that “it wasn’t bad” is mildly positive, while “it wasn’t great” is mildly negative, even though both sentences contain a negation. That’s a real step forward from older tools, which simply counted positive and negative words.
But there’s a ceiling here too, and it’s precisely what determines where automation is enough and where it isn’t. A 2020 British study found that irony and sarcasm alone can cut the accuracy of sentiment analysis by as much as half. When I first saw that figure, I had to stop and take it in. That’s not a minor deviation. It’s the difference between a tool you can trust and a tool that could easily send you in the wrong direction.
What automation can do, and where it falls short
That ceiling doesn’t mean automation is useless, though. It just means you need to know what to use it for. It excels at three things. It can process a volume no human could physically manage. It does so consistently, treating the first comment exactly the same way as the ten-thousandth. And it does it quickly, in seconds rather than weeks. For CX teams receiving hundreds of thousands of responses from support tickets, reviews and surveys, that’s a game changer.
So where does it fall short? In exactly the same places people do, just for different reasons. One 2021 study genuinely surprised me: while statistical models make more mistakes than humans overall, they tend to make them in the same places as human coders, namely on short, out-of-context or deliberately ambiguous responses. Automation doesn’t fail differently from a human. It just fails a bit more often.
And here’s a slightly sharper observation. Whenever I see tools being marketed as “AI that understands your customers better than you do”, it makes me wince. These tools aren’t a substitute for human judgement. They’re an accelerator, and they work well only when someone regularly checks and corrects them. A company that buys a tool and leaves it running unsupervised is really buying a faster route to the wrong conclusions, not better ones.
Newer models, specifically large language models (LLMs, such as GPT, which understand and generate text based on extensive training), perform somewhat better, but with one important caveat. They’re fairly good at sorting text into predefined categories. They’re much weaker at working out what those categories should be in the first place.
This tells us exactly what collaboration between people and machines should look like in practice. A human designs and tests the categories, ideally on a sample they’ve read themselves. The model then applies those categories to the rest of the data. And the human regularly comes back to check that the model isn’t drifting, because the way customers talk changes faster than anyone expects, with new words, new abbreviations and new ways of complaining.
From chart to action
Even perfectly sorted and categorised data is still just data. This is a point I’d like to dwell on, because in my view it’s where most CX projects fail. A company buys a text analytics tool, gets a beautiful dashboard with colourful pie charts splitting sentiment into positive, negative and neutral, presents it at a meeting, everyone nods, and nothing happens. I’ve seen this first-hand, and it’s one of the saddest moments in CX work: mountains of data, zero action.
A chart on its own isn’t an insight. An insight is a statement like this: customers in segment B who contacted support about a complaint in the last three months mention “long wait” three times more often than other segments, and this same group has the lowest subscription renewal rate. That’s something a product team or leadership can actually work with.
To turn a nice chart into something like that, you need three things:
First, connect your text analysis with structured data, meaning who wrote the response, what they bought and how long they’ve been a customer. A theme on its own, without knowing who’s raising it, is just a curiosity.
Second, look not only at frequency but at what’s changing over time. A theme that keeps growing, even if it doesn’t yet make up the majority of comments, is often a more important signal than one that’s large but stable.
Third, and this may be the most important, assign each insight to a specific person who takes ownership of it. A specific theme, a specific number, a specific person or team responsible for acting on it, and a deadline for reporting back. Without that, the analysis remains nothing more than interesting reading.
Pitfalls to watch out for
Even if you manage to link your analysis to your data and assign it a clear owner, there are still three traps that almost everyone falls into when they start out with text analysis. I mentioned irony and sarcasm at the very beginning, but it’s worth explaining why they’re so treacherous. It’s not just that the model doesn’t get the joke. Irony often completely flips the polarity of a sentence: the model sees positive words and scores the sentiment as positive, when the author meant the exact opposite. A partial fix is to combine sentiment with other signals, such as the score the customer gave in the same response. If someone writes an enthusiastic sentence but rates their experience one out of ten, that’s a clear warning sign not to take the text at face value.
The second trap is context, and it mainly affects short responses. The answer “expensive” to a question about product pricing is clear enough. The same word with no surrounding context, pulled out of a longer support ticket, could mean anything from the price to the waiting time to the delivery costs. Short responses are generally the biggest weakness of the whole field, because a model, just like a person, needs something to work with in order to understand, and a couple of words simply isn’t enough.
The third trap, multilingual data, is more practical than conceptual. Models today handle dozens of languages reasonably well, but not equally well. Czech also has a quirk that can trip up even a sophisticated model: idiomatic expressions built on negation. The Czech equivalent of “I can’t complain” contains a negative verb but means the complete opposite, namely that the customer is satisfied. A model trained mainly on languages with simpler negation patterns can easily score it as negative just because of the negative word. As a smaller language with rich morphology, Czech tends to be more challenging than English for this reason too: there’s less training data and more linguistic traps of this kind. In practice, if you collect feedback from several countries, it’s worth testing each language separately rather than assuming that a tool which works brilliantly on English data will be just as accurate on Czech or Slovak.
Key takeaways
I’ll admit there’s one thing I love about this whole topic: text and sentiment analysis doesn’t relieve you of the responsibility to think about your customers. It simply lets you think about far more customers at once than you ever could on your own. But the thinking itself, asking what you’ll do with a given insight and who will do it, remains a human task. Automation will find the needle in the haystack. What you do with that needle is still up to you.
Before you next click “buy” on a text analytics tool, ask yourself one simple question: who will check what the model has produced, how often, and what happens when it gets it wrong? If you don’t have an answer, you don’t have a tool yet. You just have an expensive guessing game.
