A Sentiment Score Is Not Insight: Why "78% Positive" Tells You Almost Nothing

Bottom line up front: Reducing open-text student feedback to a sentiment score—"78% of comments were positive"—recreates the exact problem that makes averaged Likert scores misleading. A percentage of positivity is not actionable: it cannot tell you what students valued, what they wanted changed, or why. Worse, off-the-shelf sentiment models systematically misread the things that matter most in student feedback—sarcasm, mixed messages, conditional praise, and domain-specific language. The point of collecting open text is to recover the meaning that numeric scales throw away. Sentiment scoring throws it away again. What you want instead is thematic analysis: structured, evidence-linked themes that name the issue and point to the fix.

Why sentiment scoring is so tempting—and so hollow

Every quality office that has ever sat on a thousand free-text comments knows the feeling: a wall of text, no time, and a dashboard that demands a number. Sentiment analysis promises rescue. Feed in the comments, get back a tidy "82% positive, 12% neutral, 6% negative," put it on a slide, move on.

The trouble is that a sentiment percentage answers a question almost nobody actually has. No programme director ever needed to know that "most comments were broadly positive." They needed to know that thirty students independently complained the lab instructions were ambiguous, that the seminar was the highlight of the module, and that the assessment deadline collided with three others. Sentiment strips precisely that out. As one survey of the field notes, free-form comments from open-ended questions are "seldom analyzed comprehensively" because they are hard to interpret and score compared with rating scales (Springer, Education and Information Technologies, 2022). Sentiment scoring does not solve that hard problem—it sidesteps it, and hands you a number that looks like analysis but contains none.

This is the same error as averaging a Likert scale, which we have written about elsewhere. A mean of 3.8 and a positivity score of 78% share a fatal property: both are one-dimensional summaries of multi-dimensional reality, and both are uninterpretable without the underlying detail they discard. You cannot act on a number that has been engineered to forget why it has the value it does.

Where sentiment models actually fail on student text

The problem is not only conceptual; the technology underperforms on the specific texture of student writing. Researchers cataloguing the limitations of sentiment analysis for student feedback point to recurring failures: identifying figurative speech such as sarcasm and irony, and handling complex constructions like double negatives, abbreviations, and unknown proper names (arXiv survey, 2020). Consider how much student feedback is exactly this:

• "Great course, if you enjoy reading 200 pages a week with no guidance." Sentiment models routinely score this positive. It is a complaint.
• "Dr. Okafor was fine, I guess." Damning with faint praise reads as neutral-to-positive. The student is disengaged.
• "Not the worst module I've taken." Two negations, faint endorsement—frequently misclassified.
• "The TA sessions saved me; the lectures were a waste of time." A single comment with opposite polarities on two different aspects, flattened into one misleading average.

That last example is the deepest problem. Most useful student comments are mixed—they praise one thing and criticise another. Document-level sentiment forces a single polarity onto a comment that contains several, which is why the field has moved toward aspect-based sentiment analysis that scores opinions about specific targets (Springer, SN Computer Science, 2022). But even aspect-based sentiment still reduces each aspect to a polarity. It tells you students felt negatively about "assessment." It does not tell you the assessment problem was deadline clustering, which is the only thing you can actually fix.

Sentiment answers "how did they feel?" Thematic analysis answers "what should we change?"

This is the core distinction. Sentiment is an affective summary. Course improvement requires a substantive one. The two are not interchangeable.

Thematic analysis—identifying recurring topics, grouping comments by the issue they raise, and linking each theme back to verbatim evidence—produces output a quality officer can act on and defend in an accreditation review. "Eleven students raised inconsistent marking between tutorial groups" is a finding. "Sentiment toward marking: 64% negative" is a thermometer reading. One leads to a change; the other leads to a meeting where everyone asks what the number means.

There is a measurement-integrity dimension too. When a sentiment percentage goes on a dashboard, it acquires false precision and becomes a target—and a single positivity number is even easier to game, and easier to over-interpret, than a mean. A two-point swing in "% positive" between cohorts will be read as a trend when it is noise. Themes resist this: a theme is either present in the evidence or it is not, and its salience is documented by the comments beneath it.

"But isn't this just sentiment analysis with extra steps? And don't LLMs make sentiment good enough now?"

This is the fair challenge. Modern large language models are far better than the lexicon-and-VADER tools the older literature critiques. Can't a good LLM just do sentiment well?

Two responses. First, even a perfect sentiment classifier still answers the wrong question. Accuracy on polarity does not rescue you from the fact that polarity is not a finding. A flawless "78% positive" is still uninterpretable; the limitation is in the metric, not the model. Second, the right use of LLMs in feedback analysis is not better sentiment—it is thematic analysis at scale: clustering comments into named issues, quantifying how many students raised each, preserving representative quotes, and flagging the rare but serious signal (a safeguarding concern, an accessibility barrier) that a positivity average would bury. The technology that makes LLMs good at sentiment is the same technology that makes them good at the more valuable task. Choosing sentiment is choosing the lesser use. (We have looked at how reliable LLM thematic analysis actually is, and what to watch for, in a separate piece.)

A second honest limitation: thematic analysis is not free of judgement. Coding decisions—where one theme ends and another begins—involve interpretation, and inter-rater reliability is a real concern. The answer is not to retreat to the false objectivity of a sentiment number, but to make the coding transparent, evidence-linked, and consistently applied.

What this looks like in practice

Koji for Education is built around this distinction. Rather than scoring comments for polarity, its AI moderator conducts a short conversational interview—when a student mentions assessment, it can ask what specifically about the assessment, turning "I didn't like the assessment" into "the deadline clashed with two other modules." Its automatic thematic analysis then groups responses across an entire cohort into named, quantified themes, each tied to the verbatim comments that support it, so a programme director sees "deadline clustering raised by 11 students" rather than a polarity bar. Because the moderation is standardised and bias-aware, the same probing is applied to every student—removing the human-coder inconsistency that makes qualitative analysis hard to trust at scale—and the six structured question types (open-ended, scale, single-choice, multiple-choice, ranking, yes/no) let you combine quantitative signal with the qualitative why.

Koji does not claim to read student minds or to remove interpretation from qualitative work. It mitigates the core failure of sentiment scoring—throwing away meaning—by preserving the link between every theme and the words students actually used. The same conversational engine powers the general Koji platform for customer and user research, where teams hit the identical wall: a sentiment dashboard that says people are 80% happy and cannot say what to build next.

The takeaway

If you have moved from averaging Likert scores to scoring sentiment, you have changed the wrapper, not the mistake. Both reduce rich, multi-dimensional feedback to a single number that cannot tell you what to do. The reason to collect open text is to recover meaning. Keep it. Ask for themes, demand the evidence beneath them, and treat any single "percent positive" figure with the same suspicion you should already have for a 3.8 average.