Topic Modeling for Open-Text Course Evaluations: LDA, STM and BERTopic Explained

In brief

Topic modeling is a family of unsupervised methods that discovers recurring themes across a large corpus of open-text comments without a human reading every one. For course evaluation, the practical choices are LDA (Latent Dirichlet Allocation, the classic bag-of-words model), STM (Structural Topic Modeling, which lets topic prevalence depend on metadata such as discipline or year), and BERTopic (an embedding-based method that captures meaning and short comments better). Real studies have run these on very large evaluation corpora — one LDA study analysed 110,420 undergraduate evaluations, and a structural-topic-modeling study analysed 286,203 comments over fourteen years — demonstrating that the approach scales. But topic models produce statistical topics, not validated constructs: they require human interpretation, careful preprocessing, and triangulation with the numeric ratings.

What the research says

Two anchor studies show what topic modeling delivers on real evaluation data.

A study published in Discover Education (2023) applied LDA to 110,420 undergraduate student-evaluation surveys collected at a US research university between 2013 and 2019. LDA treats each comment as a mixture of latent topics and each topic as a distribution over words; the model surfaced interpretable clusters covering instructor behaviours, workload, assessment, and course organisation, giving faculty a corpus-level map of what students wrote about — something no committee could produce by hand-reading a hundred thousand comments.

A second study in Education Sciences (2025) used Structural Topic Modeling (STM) on 286,203 open-ended comments spanning fourteen years at a large research university, extracting eleven latent topics and — crucially — relating each topic's prevalence to course and instructor attributes and to the accompanying quantitative ratings. STM's advantage over plain LDA is exactly this: it can estimate how the prevalence of a topic (say, "workload complaints" or "praise for feedback") varies by discipline, class size, or numeric score, turning a descriptive theme list into a testable relationship.

More recent comparative work (for example, ASEE and SciTePress conference studies, 2024) benchmarks BERTopic — which builds topics from transformer sentence embeddings rather than word counts — against LDA and k-means on student feedback. The recurring finding is that embedding-based models handle short comments and semantic nuance better (LDA struggles when a comment is only a sentence or two, which is typical of evaluations), while LDA and STM remain more transparent and easier to explain to a quality committee. There is no universal winner; the right method depends on comment length, corpus size, and how much interpretability the audience demands.

Why it matters for course evaluation in practice

Open-text comments are the richest and most under-used part of most evaluation datasets. Numeric items tell you that satisfaction dipped; comments tell you why. But at programme or institutional scale, manual thematic analysis does not scale, and selective reading invites negativity bias (the loudest complaints dominate) and cherry-picking. Topic modeling addresses three concrete QA needs:

1. Coverage at scale. A quality office can characterise an entire faculty's free-text feedback in one pass, rather than sampling a readable subset and hoping it is representative.
2. Trend detection. With STM, you can ask whether "assessment fairness" comments are rising year over year, or concentrate in particular disciplines — the kind of longitudinal signal that drives programme review.
3. Triangulation. Linking topic prevalence to numeric scores tests whether the story the numbers tell matches the story the words tell. When a department's scores are high but "workload" and "unclear expectations" topics are prevalent, that discrepancy is itself a finding.

Used well, topic modeling reframes open text from an anecdote mine into a source of programme-level evidence that stands up in an accreditation self-assessment.

Limitations and honest caveats

Topic models are powerful but easy to misuse, and a methodologically literate reader will (rightly) push on several points.

• Topics are not constructs. LDA and BERTopic return clusters of co-occurring words, not validated measures of teaching quality. Labelling a topic "assessment" is a human interpretive act, and different analysts can label the same topic differently — an inter-rater reliability problem that does not disappear just because a machine produced the clusters.
• Researcher degrees of freedom are large. The number of topics k is chosen by the analyst; preprocessing choices (stop-word removal, lemmatisation, n-grams) materially change the output; and coherence metrics only partly constrain these decisions. Two competent analysts can produce different topic solutions from the same corpus.
• Short, sparse comments break bag-of-words models. Evaluation comments are often a single line, and many students write nothing, so the corpus is short and unbalanced. LDA is particularly fragile here; embedding models help but introduce their own opacity.
• No causal or representativeness guarantee. Only students who chose to comment are in the corpus, so non-response and self-selection bias carry straight through. A prevalent topic is prevalent among commenters, not necessarily among all students.
• Language and translation. In European multilingual settings, models must be trained or validated per language; naive cross-language pooling breaks topic coherence and threatens comparability.

The honest framing is that topic modeling is a hypothesis-generating tool that must be validated against human reading, numeric evidence, and (ideally) a held-out sample — not an automated verdict on teaching quality.

How Koji incorporates this

Koji for Education treats open-text as first-class evidence and is designed to give the benefits of topic modeling while managing its limitations:

• Automatic thematic analysis clusters open-ended responses into named, human-readable themes, so a programme director sees "unclear assessment criteria" or "praise for lab support" as recurring topics across a cohort — the corpus-level map that LDA/STM produce — without running a modeling pipeline by hand.
• AI-moderated conversational interviews reduce the short-comment problem at the source. Because the interviewer asks structured follow-ups, responses are richer and more complete than a one-line box, which improves the quality of any downstream thematic clustering — directly mitigating the sparsity that breaks bag-of-words models.
• Triangulation with structured items. Koji pairs each theme with the relevant scale, single_choice and yes_no responses, so a committee can check whether a prevalent negative theme lines up with the numeric scores — operationalising the STM-style "does topic prevalence match the ratings?" test.
• Bias-aware reporting foregrounds theme prevalence and distribution rather than a handful of vivid quotes, countering the negativity bias and cherry-picking that plague manual comment reading.
• Human-in-the-loop by design. Themes are presented for review and relabelling, respecting the reality that topic labels are interpretive and require inter-rater judgement — Koji surfaces the structure but leaves the meaning to the academic reader.

These are framed as mechanisms designed to support rigorous open-text analysis, not to replace human judgement or to certify teaching quality from words alone. Koji's core research platform at koji.so applies the same conversational-interview and thematic-analysis engine to product and customer research, where making sense of thousands of open-ended responses is the same core problem.

A defensible topic-modeling workflow

Because topic models offer so many researcher degrees of freedom, the credibility of the output depends on a disciplined process rather than on the algorithm alone. A workflow that survives methodological scrutiny usually has five steps. Preprocess transparently: document stop-word lists, lemmatisation and n-gram choices, because each materially changes the topics. Select the number of topics with evidence, not taste: examine coherence and exclusivity metrics across a range of k, and report why the chosen solution was preferred. Label with more than one coder: have at least two analysts independently name the topics and record their agreement, treating labelling as the inter-rater-reliability problem it is. Validate against held-out data and human reading: check that the topics reproduce on a fresh sample and that reading a random subset of comments confirms the label. Triangulate with the numbers: relate topic prevalence to the quantitative ratings and to course metadata, as structural topic modeling is designed to do, so a theme becomes a testable relationship rather than a description. Finally, report what the model cannot see: the comments came only from students who chose to write, so a prevalent topic is prevalent among commenters, and any non-response or self-selection bias in who comments carries directly into the topic solution. Reported this way, a topic model informs programme review responsibly; a single unvalidated topic list, presented as an objective summary of teaching quality, does not.

Related Resources

• Text analytics for open-ended student comments
• Aspect-based sentiment analysis of open-text course feedback
• Using large language models to code open-text feedback
• Inter-rater reliability in thematic analysis of open text
• Thematic saturation: how many comments are enough?
• Making the most of student written comments

References

• Using topic modeling to understand comments in student evaluations of teaching (2023). Discover Education, 2, Article 51. https://doi.org/10.1007/s44217-023-00051-0
• Exploring relationships between qualitative student evaluation comments and quantitative instructor ratings: A structural topic modeling framework (2025). Education Sciences, 15(8), 1011. https://doi.org/10.3390/educsci15081011
• Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent Dirichlet Allocation. Journal of Machine Learning Research, 3, 993–1022. https://jmlr.org/papers/v3/blei03a.html
• Roberts, M. E., Stewart, B. M., & Tingley, D. (2019). stm: An R package for structural topic models. Journal of Statistical Software, 91(2), 1–40. https://doi.org/10.18637/jss.v091.i02
• Grootendorst, M. (2022). BERTopic: Neural topic modeling with a class-based TF-IDF procedure. arXiv:2203.05794. https://doi.org/10.48550/arXiv.2203.05794