Beyond "Positive or Negative": Aspect-Based Sentiment Analysis of Open-Text Course Feedback

In brief

A single sentiment score per comment is almost useless for course improvement, because most real student comments are mixed — "the lectures were excellent but the assessment was unfair and the pace too fast" is positive, negative, and negative about three different things at once. Aspect-based sentiment analysis (ABSA) decomposes each comment into the aspects it mentions (teaching, assessment, workload, materials, pace) and assigns a sentiment to each one separately. Recent studies applying ABSA to course feedback report aspect-category identification F1 scores in the low-to-mid 80s, and — more importantly — show ABSA surfaces actionable issues that purely quantitative ratings and document-level sentiment miss entirely. The honest caveats are that ABSA accuracy depends heavily on a well-defined aspect taxonomy, that sentiment models inherit domain and language bias, and that automated coding still requires human validation before it drives high-stakes decisions.

What the research says

Sentiment analysis comes in three granularities. Document-level assigns one polarity to a whole comment. Sentence-level does it per sentence. Aspect-level (ABSA) identifies the specific targets discussed and the sentiment toward each — the only granularity that matches how course feedback is actually written, because a single sentence routinely praises one thing and criticises another.

The systematic-review evidence establishes both the demand and the difficulty. Kastrati and colleagues'' systematic literature review of sentiment analysis of students'' feedback in MOOCs (2021, Frontiers in Artificial Intelligence) found that open-text feedback is rich but under-exploited, that aspect-level analysis is the most informative and least common granularity, and that the field was moving from lexicon and classical machine-learning methods toward deep learning and transformer models. A parallel study by the same group on automated analysis of open-ended student feedback (2023, Applied Sciences) combined sentiment, emotion, and cognition classification to extract more than a single polarity from each comment.

On the applied accuracy question, several recent papers report concrete numbers. Work on automatic scoring of student feedback based on aspect-level sentiment analysis (published in Education and Information Technologies, 2022) built aspect-level prediction models for course evaluation and reported strong classification performance, with one configuration using a topic dictionary plus attention reporting precision around 80%, recall around 79%, and F1 around 79%. An ABSA framework evaluated on a Coursera course-review dataset reported aspect-category identification F1 around 86%. A hybrid aspect-oriented approach (AOH-Senti, 2022, SN Computer Science) combined lexicon and learning methods to improve robustness on noisy student text. The consistent finding across these studies is not just that ABSA can be done at useful accuracy, but that aspect-based analysis identifies issues that are not apparent from quantitative ratings alone — the core practical payoff.

This sits within the wider methodological caution that NLP on student comments is not magic. As we discuss in text analytics for open-ended comments, automated methods can count and cluster but cannot by themselves understand context, sarcasm, or the institutional meaning of a theme. And as the LLM-coding agreement studies show, even large language models that agree well with human coders on average disagree on the hard, ambiguous cases that often matter most.

Why it matters for course evaluation in practice

The reason open-text feedback is so valuable — and so neglected — is captured by a familiar QA paradox: the Likert scores tell you a course scored 3.8, the open text tells you why, but nobody has time to read 400 comments per module. Document-level sentiment seems to solve the volume problem, but it does so by destroying the signal: averaging "great teacher, terrible textbook" to "neutral" throws away both actionable findings.

ABSA is the right-grained tool for three practical jobs:

• Routing the finding to the right owner. "Negative-about-assessment" goes to the module assessment lead; "negative-about-room/timetable" goes to facilities; "negative-about-pace" goes to the lecturer. A document-level score cannot route anything.
• Separating the instructor from the course. Students conflate the two, and so do raw scores. Aspect separation lets a programme see that a poorly rated module had excellent teaching but a broken assessment design — a distinction that changes whether you retrain a person or redesign a unit.
• Counting themes honestly. ABSA lets you say "31% of comments raised workload, and of those 78% were negative," which is far more decision-useful than an average star rating — and it interacts with thematic saturation (how many comments you need before themes stabilise) and inter-rater reliability (whether your coding — human or machine — is consistent).

Crucially, ABSA also disciplines the human reading of comments. Left to themselves, readers fall prey to negativity bias, fixating on the one abusive comment. A per-aspect tally — "two hostile comments, both about grading; eighteen constructive comments, mostly about pacing" — reframes the same data toward what is representative and fixable.

Limitations and honest caveats

ABSA is powerful but not turnkey, and a critical reader should hold it to account on several fronts:

• Aspect taxonomy is a design choice, not a discovery. ABSA performance depends on a well-defined set of aspect categories. Too coarse ("teaching") and you lose the routing benefit; too fine and the model has too few examples per class. The taxonomy encodes assumptions about what matters, and a bad taxonomy produces confidently wrong structure.
• Reported accuracy is dataset-specific. An F1 of 86% on Coursera English reviews does not transfer to short, code-switched comments from a multilingual European cohort. Sentiment and aspect models inherit the domain, language, and demographic biases of their training data, and degrade on out-of-distribution text. Cross-language equivalence (see questionnaire translation) is a live problem for multilingual ABSA.
• Sentiment is not severity or actionability. A calmly worded "the deadlines clustered badly" is more actionable than an emphatic "loved it!", but a polarity model scores the second as the stronger signal. Sentiment intensity and institutional importance are different things.
• Sarcasm, negation, and context remain hard. "Oh great, another 9am exam" is positive by surface lexicon and negative in meaning. Modern transformers handle some of this; none handle all of it.
• Automation invites over-trust. A clean dashboard implies precision the underlying classifier may not have. ABSA output should be treated as a triage and prioritisation layer that a human validates, not as a verdict — especially before it feeds any decision about a person.

How Koji incorporates this

Koji for Education applies aspect-aware analysis as a core part of how it turns open text into usable findings, while keeping the human in the loop the limitations demand:

• Aspect-aware thematic analysis. Koji automatically clusters open-ended responses into themes (aspects) and reports sentiment per theme rather than as a single comment-level score — so a programme sees "assessment: mostly negative; teaching clarity: mostly positive" instead of an averaged blur.
• Probing generates cleaner aspect signal at the source. Because Koji collects feedback through AI-moderated conversational interviews rather than a single free-text box, it can ask follow-ups that pin down which aspect a vague comment refers to ("you said the course was stressful — was that the workload, the pace, or the assessment?"). This produces text that is already aspect-resolved, reducing the burden on downstream classification and its error rate.
• Representative quotes, not just polarity. For each theme Koji surfaces verbatim example responses, so an evaluation committee reads the actual student voice behind a sentiment tally — guarding against the over-trust and context-loss that pure scoring invites.
• Quality scoring and triangulation. Koji scores response quality and triangulates the open-text themes against the structured scale and single_choice answers, flagging where the numbers and the narrative diverge rather than presenting either in isolation.
• Designed to assist, not adjudicate. Koji frames aspect-based output as a prioritisation and routing layer for human review, explicitly cautioning against treating automated sentiment as a final judgement — consistent with the validation requirement the research insists on.

The same conversational-analysis engine powers Koji''s core research platform at [koji.so], where aspect-based analysis of open-ended responses is applied to product and customer feedback using the identical "resolve the aspect, score per aspect, show the quote" workflow.

Related Resources

• Text Analytics for Open-Ended Student Comments: What NLP Can and Cannot Tell You
• Can a Large Language Model Code Your Open-Text Feedback?
• How Many Open-Text Comments Are Enough? Thematic Saturation
• How Reliable Is Your Coding of Open-Text Feedback? Inter-Rater Reliability
• Why One Cruel Comment Outweighs Twenty Kind Ones: Negativity Bias
• What Can Open-Text Student Comments Tell You That Likert Scores Cannot?

References

• Kastrati, Z., Dalipi, F., Imran, A. S., Pireva Nuci, K., & Wani, M. A. (2021). Sentiment Analysis of Students'' Feedback With NLP and Deep Learning: A Systematic Mapping Study. Applied Sciences, 11(9), 3986. https://doi.org/10.3390/app11093986
• Dalipi, F., Zdravkova, K., & Ahlgren, F. (2021). Sentiment Analysis of Students'' Feedback in MOOCs: A Systematic Literature Review. Frontiers in Artificial Intelligence, 4, 728708. https://doi.org/10.3389/frai.2021.728708
• Lalata, J. P., Gerardo, B., & Medina, R. (2022). Automatic scoring of student feedback for teaching evaluation based on aspect-level sentiment analysis. Education and Information Technologies, 27, 7755–7777. https://doi.org/10.1007/s10639-022-11151-z
• Sindhu, I., et al. (2022). AOH-Senti: Aspect-Oriented Hybrid Approach to Sentiment Analysis of Students'' Feedback. SN Computer Science, 3, 471. https://doi.org/10.1007/s42979-022-01611-1
• Liu, B. (2012). Sentiment Analysis and Opinion Mining. Synthesis Lectures on Human Language Technologies. Morgan & Claypool. https://doi.org/10.2200/S00416ED1V01Y201204HLT016