Can AI Reliably Analyze Thousands of Open-Text Student Comments?

Bottom line up front: Large language models can thematically analyse thousands of open-text student comments with moderate-to-substantial agreement with trained human coders and near-perfect self-consistency — good enough to surface patterns no human team has the hours to find manually, but not good enough to be left unsupervised on high-stakes decisions. The honest position is neither "AI replaces qualitative analysis" nor "AI cannot be trusted." It is: AI makes the previously unread readable, provided you validate it and keep a human in the loop.

The problem AI is actually solving

Course evaluations have a dirty secret. The numbers get tabulated and benchmarked; the open-text comments — usually the richest, most actionable data — get skimmed by one busy coordinator, if read at all. A programme with 1,500 students and three open questions per survey generates 4,500 comments a cycle. No quality-assurance office reads, codes, and cross-tabulates that by hand every term. So the qualitative signal, the part that tells you why a score moved, is routinely thrown away.

This is the gap automated thematic analysis addresses. The question is not whether AI is perfect at it — humans are not perfect at it either — but whether AI is reliable enough to be useful, and where its limits sit.

What the evidence shows on reliability

The right yardstick is inter-rater reliability: how well does an LLM agree with trained human coders, measured with Cohen's or Fleiss' kappa? Kappa corrects for chance agreement; by the widely used Landis and Koch (1977) benchmarks, 0.41–0.60 is "moderate," 0.61–0.80 "substantial," and above 0.80 "almost perfect." Two human coders on messy real-world text frequently land in the moderate-to-substantial band themselves, which is the fair comparison — not an imaginary perfect human.

Recent studies, mostly in health-care and social-science qualitative research, are converging:

• A 2024 study in the Journal of Medical Internet Research comparing GPT-4 with human researchers on qualitative health-care data found moderate agreement (κ = 0.40), with GPT-4 reliably identifying major themes while humans surfaced a richer set of sub-themes (JMIR, 2024).
• Across several comparisons, human–LLM Fleiss' kappa ranged from 0.44 to 0.50 (mean ≈ 0.46) — squarely "moderate."
• Newer models do better on structured tasks: GPT-4o reached substantial agreement (κ = 0.61–0.65) on multi-label thematic categorisation in a 2025 JMIR AI analysis (JMIR AI, 2025).
• Most striking: when the same model coded the same data three times, self-consistency was κ ≈ 0.87 ("almost perfect"). The machine is far more internally consistent than a fatigued human reading comment 3,000.

Put together, the picture is sober and encouraging. LLMs match humans tolerably well on identifying the main themes, are weaker on nuance and rare sub-themes, and are dramatically more consistent run-to-run. That is exactly the profile you want for a first-pass instrument over thousands of comments — and exactly why you still want a human reviewing the output.

But doesn't AI hallucinate and miss the nuance?

This is the serious objection, and pretending otherwise would forfeit the credibility this whole argument rests on. Four real limitations:

1. Hallucination and over-claiming. An LLM can assert a theme that the data does not support, or attach a confident summary to a pattern that is thin. Mitigation: ground every theme in retrievable verbatim quotes, and report prevalence (how many comments) rather than vibes.
2. Loss of the long tail. The evidence is consistent that humans find more sub-themes. The single furious comment about an inaccessible lab, or the quiet one about a disability accommodation, is precisely what you must not average away. Mitigation: never let thematic rollups replace reading flagged outliers; surface them.
3. Prompt and model sensitivity. Results shift with prompt wording and model version, which threatens reproducibility. Mitigation: fixed, documented prompts; versioned models; periodic re-validation against a human-coded sample.
4. Bias. Models can carry biases that interact badly with feedback already known to be biased — for example, comments that themselves reflect gender or accent bias against instructors. Mitigation: bias-aware design, human review of staff-facing outputs, and never treating themes as verdicts about a person.

None of these is a reason to abandon automated analysis. Each is a reason to bound it: validate against human coding, keep quotes attached, escalate outliers, and keep humans deciding anything that matters. "Mitigates and surfaces," not "eliminates and decides."

A defensible workflow

The methodologically honest way to use LLMs on student feedback looks like this:

• Validate up front. Have humans code a representative sample, run the model on the same sample, and report the kappa. If it is below "moderate," fix the prompt or the taxonomy before trusting it at scale.
• Theme at scale, with evidence. Let the model code the full corpus, but require every theme to cite verbatim comments and a prevalence count.
• Quality-score the inputs. Not every comment carries signal; flagging low-information or off-topic responses improves the analysis.
• Escalate the tail. Route outliers, safeguarding concerns, and abusive comments to humans rather than smoothing them into a theme.
• Re-validate periodically. Models and student populations drift; a standing human-coded benchmark catches it.

How to read a vendor's reliability claims

Because the evidence is nuanced, the way a vendor talks about AI analysis tells you most of what you need to know. Demand three things, and be wary if you do not get them.

First, a number, not an adjective. "Highly accurate" is marketing; a reported agreement statistic — Cohen's or Fleiss' kappa against human coders on a comparable corpus — is evidence. Ask what the figure is and how it was obtained. A vendor confident in their analysis can tell you.

Second, traceability. Every theme in a report should link to the verbatim comments behind it and a prevalence count. If you cannot click from "students found assessment unclear" to the actual sentences, you cannot audit the claim — and neither can an external reviewer.

Third, versioning and re-validation. Models change, and a result generated on one model version may not reproduce on the next. A serious tool documents the model and prompt and re-validates against a human-coded sample on a schedule, so your time series is not silently broken by an upgrade.

These are not exotic asks. They are the qualitative-research equivalents of reporting your method, showing your data, and stating your instrument. A tool that resists them is asking for trust it has not earned.

Consistency is underrated

It is worth dwelling on the κ ≈ 0.87 self-consistency figure, because it inverts a common worry. The standard fear about AI is that it is erratic; the evidence says the opposite — on the same data, a model returns nearly the same coding every time, far more reliably than a human team whose judgements drift with fatigue, mood, and who happens to be coding that week. For a longitudinal evaluation programme, that stability is not a footnote — it is what makes term-on-term comparison meaningful. If your instrument wobbles, you cannot tell a real change in student experience from noise in the coders. The caveat, already noted, is that consistency is not correctness: a model can be reliably wrong, which is exactly why the human-coded validation sample is non-negotiable. But once validated, the run-to-run stability you gain is a genuine methodological advantage that manual coding, for all its richness, struggles to match at scale.

Where Koji fits

This workflow is, in effect, Koji's design. Koji for Education runs AI-moderated conversational interviews that elicit far richer open text than a comment box, then applies automatic thematic analysis with verbatim evidence and prevalence, quality scoring to separate signal from noise, and bias-aware, standardised moderation so the elicitation itself is consistent across students. Critically, programme- and institution-level reporting is decision support for humans — committees and teaching-and-learning centres — not an automated judgement about any instructor. The point is to make 4,500 comments legible, not to outsource your judgement to a model.

Because the underlying interview-and-analysis engine is general, teams that also do customer, employer, or staff research can run the same approach on the main Koji platform — one validated, evidence-attached method across institutional and operational research.

AI does not make qualitative rigour obsolete; it makes it scalable — if, and only if, you keep the human in the loop and the quotes on the page. If you want to see thematic analysis that shows its working, explore Koji for Education.