Semantic Text Embeddings for Open-Text Course Feedback

Open-text comments are the richest part of a course evaluation and the least used, because reading them does not scale. The usual automated responses — count keywords, run sentiment, fit a topic model — each throw away most of the meaning. "The lectures dragged" and "pacing was far too slow" share no keywords yet say the same thing; a keyword counter treats them as unrelated, and even a topic model only tells you they belong to a "pacing" cluster without letting you ask precise questions of the text. Text embeddings are the representation that fixes this: they turn each comment into a point in a high-dimensional space where meaning, not vocabulary, determines distance. That single move enables semantic search, near-duplicate detection, coverage measurement, and reliable routing of comments to actions — capabilities distinct from topic modelling or sentiment analysis.

The short answer (BLUF)

An embedding is a fixed-length numeric vector that represents a comment's meaning, learned so that semantically similar texts land close together (measured by cosine similarity) even when they share no words. For open-text course feedback this unlocks four things a keyword or topic-model pipeline cannot do well: (1) semantic search — retrieve every comment about "assessment fairness" regardless of phrasing; (2) near-duplicate and redundancy detection — collapse fifty ways of saying "great lecturer" so you do not over-count a single sentiment; (3) coverage and novelty measurement — quantify how much of this term's feedback is genuinely new versus a repeat of last term; and (4) routing — match each comment to the closest predefined action category. Embeddings are a representation, not an interpretation: they tell you what is close to what, and you still need human judgement to decide what the clusters mean.

What the research says

The foundational idea is the distributional hypothesis — words that occur in similar contexts have similar meanings — operationalised at scale by Mikolov and colleagues (2013), whose word2vec embeddings placed individual words in a vector space where semantic relationships became geometric. The leap to whole sentences and comments came with contextual models: Devlin and colleagues (2019) introduced BERT, whose representations depend on surrounding context, and Reimers and Gurevych (2019) built Sentence-BERT, which produces sentence embeddings that can be compared directly with cosine similarity — turning a semantic-similarity query that was previously prohibitively expensive into a single fast vector comparison. Their key contribution was practical: earlier BERT required feeding both texts through the network together for every comparison, which is infeasible over thousands of comments; Sentence-BERT produces one reusable vector per text, so similarity across an entire evaluation corpus becomes tractable.

Applied to student feedback specifically, the evidence base is now substantial. Sliusarenko, Clemmensen, and Ersbøll's text-mining study of course evaluations extracted the main topics of students' open-ended comments and related them quantitatively to the numeric scores, demonstrating that written feedback carries structured, analysable signal beyond the ratings. More recent work applies word embeddings and transformer models (BERT, RoBERTa, XLNet) to opinion mining and topic classification of course reviews, and text-analytics pipelines have been built specifically to extract actionable improvement suggestions from students' comments — a task that depends on matching free text to intent, which is exactly what embeddings do well. The through-line: embedding-based representations consistently outperform bag-of-words approaches when the goal is to group, retrieve, or route feedback by meaning.

Why it matters for course evaluation in practice

You stop over-counting and under-counting. Keyword dashboards over-count the loud vocabulary and miss the paraphrases; embeddings let you count distinct concerns rather than distinct words. Near-duplicate detection means twenty rephrasings of one complaint register as one strong signal, not twenty weak ones — and one carefully argued minority point is not buried under repetition.

You can search the corpus like a database. A quality officer asking "did anyone raise concerns about the lab safety briefing?" can retrieve every semantically relevant comment in seconds, instead of hoping the right keyword was used. This is the difference between feedback you can interrogate and feedback you merely archive.

You can measure coverage and novelty across terms. By embedding this term's comments and comparing them to last term's, you can quantify how much feedback is genuinely new — a direct, defensible answer to "did our intervention change what students talk about?" that neither sentiment nor a fresh topic model gives cleanly.

You can route feedback to owners. Matching each comment to the nearest predefined action category (assessment, workload, accessibility, resources) turns a wall of text into a triaged queue — the operational heart of closing the feedback loop.

Limitations and honest caveats

Embeddings encode meaning as the training data represented it, and they inherit its biases: models can under-represent the phrasing of non-native English speakers, encode demographic stereotypes, and treat domain jargon unpredictably. Cosine similarity is a similarity, not a truth — two comments can be "close" for reasons irrelevant to your question (shared politeness formulae, shared course name), so thresholds must be validated against human judgement, not assumed. Embeddings also flatten negation and stance: "the feedback was timely" and "the feedback was not timely" can sit uncomfortably close, so embeddings should complement, not replace, aspect-based sentiment when polarity matters. General-purpose models may miss institution-specific meaning without adaptation, and the whole approach is opaque — a vector is not an explanation, and a cluster is not a theme until a human names it and checks that the naming holds. Finally, small classes generate few comments, where embedding-based clustering is unstable and manual reading remains both feasible and preferable. These are reasons to treat embeddings as an assistive representation under human oversight, not an autonomous analyst.

How Koji incorporates this

Koji's open-text analysis is built on semantic representations rather than keyword matching, which is what lets it group paraphrased feedback, surface a well-argued minority view, and avoid the double-counting that inflates keyword dashboards. Because Koji embeds comments into a semantic space, its automatic thematic analysis clusters by meaning and supports semantic search across a course's, module's, or programme's entire feedback corpus — so a quality-assurance officer can ask a question of the text and get every relevant comment back, not just the ones that used the expected word. The same representation powers near-duplicate detection (so twenty rephrasings of one issue register as one strong theme with its true weight) and coverage comparison across cohorts and terms, feeding Koji's closing-the-loop action tracking by routing each comment to the most relevant improvement category. Critically, Koji pairs embeddings with its AI-moderated conversational interview: when a comment is terse or ambiguous, the interview has already probed for specifics, so the text being embedded is richer and the semantic clusters are sharper than anything recoverable from a one-line box. And because embeddings can misread negation and stance, Koji keeps polarity as a separate, bias-aware signal rather than trusting proximity alone. The identical embedding engine runs in Koji's core research platform at koji.so, where product and customer teams face the same need to search, deduplicate, and route large volumes of open-ended responses. Koji frames all of this as designed to mitigate the limits of keyword and topic-only analysis under human oversight — not to replace the analyst who decides what a cluster means.

Frequently asked questions

What is a text embedding, in plain terms?

It is a list of numbers that represents the meaning of a piece of text, produced so that texts with similar meaning have similar number-lists. You compare two embeddings with cosine similarity: a high value means the comments mean roughly the same thing, even if they share no words.

How is embedding-based analysis different from topic modelling?

Topic modelling (LDA, BERTopic) groups comments into a set of discovered topics; embeddings give you the underlying meaning-space itself, which you can then search, deduplicate, measure for novelty, and route — operations a topic model does not directly provide. Many modern topic models actually run on top of embeddings; the embedding is the more general representation.

Do embeddings understand sentiment?

Not reliably. Embeddings capture topical and semantic similarity but can place "was timely" and "was not timely" close together because they flatten negation. For polarity you should keep a dedicated sentiment or aspect-based sentiment step alongside the embeddings.

Can embeddings handle comments in multiple languages?

Multilingual embedding models can place equivalent meanings from different languages near each other, which helps European institutions with mixed-language cohorts. Quality varies by language and model, so links across languages should be spot-checked against human judgement before they drive reporting.

Will embeddings just invent themes that are not there?

Embeddings do not invent themes — they measure proximity. Themes appear only when someone (or a clustering step) draws boundaries in the space and names the groups. That naming is a human interpretive act that must be validated; a cluster is a hypothesis, not a finding.

Are embeddings safe for small classes?

For a handful of comments, embedding-based clustering is unstable and manual reading is both feasible and more trustworthy. Embeddings earn their value at scale — dozens to thousands of comments — where reading everything is impossible.

Related resources

References

Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space. arXiv:1301.3781. https://arxiv.org/abs/1301.3781
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of NAACL-HLT 2019, 4171–4186. https://doi.org/10.18653/v1/N19-1423
Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence embeddings using Siamese BERT-networks. Proceedings of EMNLP-IJCNLP 2019, 3982–3992. https://doi.org/10.18653/v1/D19-1410
Sliusarenko, T., Clemmensen, L. H., & Ersbøll, B. K. (2013). Text mining in students' course evaluations: Relationships between open-ended comments and quantitative scores. Proceedings of the 5th International Conference on Computer Supported Education (CSEDU).
Gottipati, S., Shankararaman, V., & Lin, J. R. (2018). Text analytics approach to extract course improvement suggestions from students' feedback. Research and Practice in Technology Enhanced Learning, 13, 6. https://doi.org/10.1186/s41039-018-0073-0

Beyond Keyword Counts: Semantic Text Embeddings for Searching, Deduplicating, and Mapping Open-Text Feedback