Faithfulness and Hallucination in LLM Course-Feedback Summaries

The short answer

Large language models are now routinely used to summarise the free-text comments in course evaluations — turning hundreds of student remarks into a tidy paragraph of "key themes." The problem, well documented in the summarisation literature, is that abstractive models are prone to hallucination: they generate fluent, plausible statements that are not supported by the source text. In a QA report, a hallucinated theme can send a programme chasing a problem that does not exist, and a fabricated or misattributed quote can unfairly damage an instructor. Faithfulness — every claim in the summary being traceable to something a student actually wrote — is therefore not a nicety but the core quality requirement of any AI feedback pipeline.

BLUF: Abstractive summarisation research (Maynez et al., 2020) found that neural summarisers hallucinate frequently, and that most hallucinations are extrinsic — assertions that cannot be verified from the source at all, rather than mere distortions of it. Standard overlap metrics such as ROUGE correlate poorly with human judgements of faithfulness; entailment-based checks do better. For course evaluation this means an AI summary must be grounded and quote-anchored: every theme tied to verbatim comments, every claim checkable against the source, and a human able to audit the chain. Treat an unanchored AI summary of student feedback as an unverified draft, not evidence.

What the research says

Maynez, Narayan, Bohnet & McDonald (2020), "On Faithfulness and Factuality in Abstractive Summarization" (ACL 2020) is the anchor. Running a large-scale human evaluation across several neural abstractive summarisation systems, they found that the models hallucinated substantial amounts of content in all systems, and — the key distinction — that the majority of hallucinations were extrinsic: statements not entailed by, and not even derivable from, the input document. They also showed that ROUGE and similar n-gram-overlap metrics are poor proxies for faithfulness, whereas textual-entailment-based measures aligned far better with human judgements of whether a summary was faithful. A practical corollary from their work: models with stronger language priors were more fluent but not automatically more faithful, so fluency must never be read as accuracy.

Ji et al. (2023), "Survey of Hallucination in Natural Language Generation" (ACM Computing Surveys) gives the vocabulary the field now uses. It separates intrinsic hallucination (output that contradicts the source) from extrinsic hallucination (output the source can neither confirm nor deny), and catalogues causes spanning data (source-reference divergence) and modelling (exposure bias, decoding strategies). The taxonomy matters for course evaluation because the two failure modes need different guards: intrinsic errors are caught by consistency checks against the source; extrinsic ones are caught by refusing to assert anything not grounded in a quote.

Two corroborating works sharpen the operational picture. Fabbri et al. (2021), "SummEval" (TACL) re-evaluated summarisation metrics against careful human ratings and found that automatic metrics correlate weakly with human judgements of factual consistency and relevance — you cannot certify faithfulness with ROUGE alone. And Pagnoni, Balachandran & Tsvetkov (2021), the FRANK benchmark (NAACL), built a typology of factual-error types in generated summaries and showed that different systems fail in systematically different ways, reinforcing that "is this summary faithful?" is a measurable, not merely rhetorical, question.

The literature has moved on to instruction-tuned LLMs, which hallucinate less than the 2020-era models — but "less" is not "never," and abstractive systems continue to fabricate under distribution shift, long inputs, and sparse or contradictory sources, all of which describe a pile of open-text course comments.

Why it matters for course evaluation in practice

Open-text comments are the richest and most trusted part of an evaluation — and the part most often now passed through an LLM. The specific hazards:

Invented themes. Ask a model to "list the top themes" and it will always return a list, even when a theme is thin or absent. An extrinsic hallucination here manufactures a QA priority out of nothing — the machine-generated cousin of the availability heuristic, where a vivid impression is mistaken for a pattern.
Fabricated or misattributed quotes. A summary that "quotes" a student can invent wording or attach a real comment to the wrong course or instructor. In a personnel context this is not a data-quality issue but a fairness and potentially defamation issue.
Phantom prevalence. LLMs freely write "many students felt…" without any count behind it. Uncorrected, this fuses hallucination with denominator neglect, giving unquantified impressions the authority of numbers.
Sentiment inversion. Intrinsic hallucination can flip a hedged or sarcastic comment into a clear positive or negative, corrupting any downstream aspect-based sentiment reading.

Because these outputs feed accreditation self-evaluations and, increasingly, are re-summarised by the AI assistants staff consult, an unfaithful summary does not stay contained — it propagates.

Limitations and honest caveats

The anchor studies predate current LLMs. Maynez et al. evaluated 2020-era systems; modern instruction-tuned models hallucinate less. The direction of the finding is robust and repeatedly replicated, but exact rates should not be quoted as if they describe today's GPT-class models.
Faithfulness is not the only goal. A perfectly faithful summary can still be useless if it is unrepresentative — over-weighting a loud minority. Grounding fixes fabrication, not sampling bias; the two must be addressed separately.
Entailment checks are themselves imperfect. Automated faithfulness verifiers (NLI models, LLM-as-judge) have their own error rates and can miss subtle misattribution. They reduce risk; they do not certify truth.
Extractive is safer but poorer. Pure verbatim extraction cannot hallucinate but also cannot synthesise. The realistic target is grounded abstraction — synthesis that is forced to cite — not a retreat to extraction.
Human review has limits at scale. Auditing every claim against thousands of comments is exactly why teams reach for AI in the first place; the guard rails must make spot-checking cheap, because exhaustive checking defeats the purpose.

How Koji incorporates this

Koji treats faithfulness as an engineering requirement of its automatic thematic analysis, not an assumed property of the model.

Quote-anchored themes. Every theme Koji surfaces is linked to the verbatim student comments that support it. An analyst can click a theme and read the exact source text, so a claim with no anchor is visibly unsupported — the direct operational answer to extrinsic hallucination.
Grounded, not free-form, generation. Koji's summaries are constrained to the provided comments and are designed to abstain rather than embellish when evidence is thin, targeting the "always returns a list" failure mode.
Prevalence from counts, not prose. Quantity language ("most", "many") is backed by actual response counts and proportions in Koji, not generated by the model — separating measured prevalence from narrative and mitigating the phantom-prevalence and denominator-neglect trap.
Faithfulness verification and flags. Consistency checks (entailment-style verification that each stated claim is supported by cited comments) flag summary statements that are not grounded, so reviewers audit the exceptions rather than the whole corpus.
Conversational depth to reduce the source problem. Much hallucination originates in thin, ambiguous source text. Koji's AI-moderated conversational interviews elicit fuller, less ambiguous responses in the first place — a better source is easier to summarise faithfully. This is framed as mitigation, not a guarantee: Koji is designed to make unfaithful output detectable and rare, not to claim it is impossible.

Koji's core research platform at koji.so applies the same quote-anchored, grounded-summarisation discipline to product and customer feedback, where a hallucinated "theme" can just as easily misdirect a roadmap.

Related resources

Frequently asked questions

What is the difference between intrinsic and extrinsic hallucination?

Intrinsic hallucination is output that contradicts the source — for example flipping a negative comment to a positive one. Extrinsic hallucination is output the source can neither confirm nor deny — a theme or claim with no basis in any comment. Maynez et al. (2020) found the extrinsic type was the more common in abstractive summaries, and it is the harder one to catch because there is nothing in the source to compare against.

Can I trust an AI summary of my course comments?

Only if it is grounded. A faithful pipeline links every theme and claim to the verbatim comments that support it, so you can audit the chain. Treat an unanchored, free-form summary as an unverified draft, especially before using it in personnel or accreditation decisions.

Do newer LLMs still hallucinate on feedback summaries?

Less than 2020-era models, but not never. Abstractive systems continue to fabricate under long inputs, ambiguous or contradictory comments, and requests to always produce a fixed number of themes — conditions that describe a typical set of open-text course comments.

Why is ROUGE not enough to check faithfulness?

ROUGE measures n-gram overlap with a reference summary, not whether claims are supported by the source. Maynez et al. (2020) and the SummEval study (Fabbri et al., 2021) both found overlap metrics correlate weakly with human judgements of factual consistency; entailment-based checks perform better.

How do I stop an AI from inventing prevalence like most students?

Separate measurement from narrative. Quantity statements should be produced from actual response counts and proportions, not written by the language model. Grounding the prose in counts prevents unquantified impressions from acquiring the authority of numbers.

Is extractive summarisation the safe answer?

Extractive summaries cannot hallucinate because they only copy source text, but they cannot synthesise across comments either. The practical target is grounded abstraction — synthesis that is required to cite its supporting quotes — rather than abandoning synthesis altogether.

References

Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020). On faithfulness and factuality in abstractive summarization. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 1906–1919. https://aclanthology.org/2020.acl-main.173/
Ji, Z., Lee, N., Frieske, R., et al. (2023). Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12), Article 248. https://doi.org/10.1145/3571730
Fabbri, A. R., Kryściński, W., McCann, B., Xiong, C., Socher, R., & Radev, D. (2021). SummEval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9, 391–409. https://doi.org/10.1162/tacl_a_00373
Pagnoni, A., Balachandran, V., & Tsvetkov, Y. (2021). Understanding factuality in abstractive summarization with FRANK: A benchmark for factuality metrics. Proceedings of NAACL-HLT 2021, 4812–4829. https://doi.org/10.18653/v1/2021.naacl-main.383

When the AI Summary Says Something No Student Did: Faithfulness and Hallucination in LLM Course-Feedback Summaries