It's Not What They Say, It's How: Discourse Analysis of Open-Text Course Feedback

In brief

Discourse analysis treats a student's written comment not as a transparent report of their satisfaction but as a piece of language doing work — positioning the writer, drawing on shared cultural repertoires, and constructing a version of the course. Where thematic coding asks "what topics recur?", discourse analysis asks "how is this said, what is it doing, and what does the phrasing take for granted?" Potter and Wetherell (1987) and Gee (2014) give the method its shape. It is powerful for exposing the assumptions baked into feedback — the "student-as-consumer" voice, the gendered vocabulary of praise — but it is labour-intensive, interpretive, and does not scale to counting.

What the research says

Discourse analysis is not one method but a family of approaches united by a single premise: language is constructive and functional, not a neutral window onto an inner state. Potter and Wetherell (1987), in the founding text of the discursive-psychology strand, argue that when a person writes or speaks they are not simply venting a pre-formed attitude; they are assembling an account that performs an action — justifying, blaming, excusing, aligning — and that draws on culturally available interpretative repertoires: recognisable clusters of terms, metaphors and figures of speech through which a version of reality is built. Wetherell (1998) develops this with the idea of subject positions — the identities a stretch of talk makes available to its speaker — and shows how a single comment can take up shifting, even contradictory, positions.

Gee (2014) provides the most widely used practical toolkit, framing analysis around the "building tasks" language performs: any utterance simultaneously builds significance (what is made to matter), activities, identities, relationships, and connections. His method directs the analyst to ask, of every clause, why this way and not another? Fairclough's critical discourse analysis adds the dimension of power: how institutional and ideological relations are reproduced in ordinary text, so that a phrase like "value for money" in a course comment is read as importing a whole marketised framing of education. Rogers, Malancharuvil-Berkes, Mosley, Hui and O'Garro Joseph (2005), reviewing critical discourse analysis across educational research, document both its rapid uptake and its methodological unevenness — a field rich in insight but inconsistent in how systematically the analysis is actually done.

That last point is sharpened by Antaki, Billig, Edwards and Potter (2003), whose influential critique lists six ways discourse "analysis" often fails to be analysis at all: under-analysis through summary, through taking sides, through over-quotation, through spotting isolated features, and so on. Their warning is the discipline's own quality gate: quoting a few comments and paraphrasing them is not discourse analysis; the method requires systematic attention to how the text is constructed and what it accomplishes. Across these sources the consistent claim is that open-text feedback carries a layer of meaning — assumptions, positionings, power relations — that neither a star rating nor a frequency count of themes can reach.

Why it matters for course evaluation in practice

Open-text comments are the richest part of most evaluations and the most under-analysed. Standard practice either ignores them or reduces them to sentiment and theme counts. Discourse analysis recovers what those reductions discard: the framing students bring to their judgement.

Consider three practical uses. First, surfacing the consumer frame. When comments are dense with "I paid for this," "value for money," and "customer service," a discourse reading identifies an interpretative repertoire that treats the course as a purchased product — a framing with documented effects on how students evaluate, and one a programme may want to notice and address rather than simply score. This connects directly to the student-written-comments evidence about what open text uniquely reveals. Second, exposing bias in the language of praise. The finding that men are called "brilliant" and women "caring" is, at root, a discourse-analytic observation about the repertoires available for describing male and female instructors; reading comments this way makes the gendered vocabulary of evaluation visible in a way a sentiment score never will. Third, understanding how blame and credit are assigned. Discourse analysis reveals whether students position themselves as active agents in their own learning or as passive recipients acting on the instructor — a distinction with real pedagogical meaning that is invisible to counting.

Used this way, discourse analysis is a complement to the scalable qualitative methods already in the toolkit — the framework method, grounded theory, and narrative inquiry — each of which reads open text through a different lens. Where those often aim at what students say and the explanations behind it, discourse analysis targets how the saying is done and what it presupposes.

Limitations and honest caveats

Discourse analysis is the least scalable and most interpretively demanding method in the qualitative repertoire, and a credible QA process is honest about that.

It does not count, and it does not generalise in the statistical sense. A discourse analysis of forty comments produces an interpretation of how certain accounts are constructed, not an estimate of how many students hold a view. It cannot answer "what proportion of the cohort was dissatisfied?" and should never be reported as if it could. It is a tool for understanding meaning, not measuring prevalence.

It is labour-intensive and expertise-dependent. Doing it well requires training; the Antaki et al. (2003) critique exists precisely because so much published work labelled "discourse analysis" is really just illustrated summary. For a busy QA office, rigorous discourse analysis of a whole institution's comments every term is simply infeasible, and pretending otherwise produces the under-analysis those authors warn about.

It is interpretive and analyst-dependent. Two skilled analysts can read the same comment differently, and there is no inter-rater kappa to adjudicate — the method does not aspire to that kind of reliability. This is a feature, not a bug, of an interpretive approach, but it means findings must be argued and evidenced from the text, and treated as a defensible reading rather than the answer.

It can over-read. Not every phrase is ideologically loaded; sometimes "the lectures were clear" just means the lectures were clear. The discipline's own guard against over-interpretation is systematic analysis grounded in the data, and the temptation to find a marketised subtext in every sentence should be resisted.

Because of all this, discourse analysis belongs on a purposive sample — a difficult module, a contested redesign, a set of comments that surprised the team — not on the full firehose of institutional feedback.

How Koji incorporates this

Koji does not automate discourse analysis — no system credibly can, since the method is interpretive by definition — but it is designed to make rigorous, human discourse analysis feasible by getting richer text and organising it for a human analyst. Because Koji's evaluations are AI-moderated conversations rather than one-shot comment boxes, the open text students produce is fuller and more accountable: when a student writes "it wasn't worth it," the moderator's follow-up probe elicits the elaboration a discourse analyst needs to see how the account is built, rather than leaving a bare fragment. That depth is exactly what the Antaki et al. critique demands as the raw material for real analysis.

The platform's automatic thematic analysis and semantic search then serve as a triage layer for the human analyst: a QA officer can pull the purposive sub-corpus discourse analysis requires — every comment invoking cost or value, or all feedback on a contested module — instead of reading everything. Koji's structured export keeps each comment linked to its full conversational context and metadata, so the analyst studies the exchange, not an orphaned sentence. Throughout, the framing is that Koji is designed to support expert interpretive analysis of a focused sample, not to replace it with a score — the platform surfaces and organises the discourse; a trained human reads it. The same conversational engine and semantic retrieval power Koji's core research platform at koji.so, where UX researchers apply identical close-reading of how customers frame a product to a purposively selected slice of interviews.

Frequently asked questions

How is discourse analysis different from thematic analysis?

Thematic analysis identifies what topics recur across comments and roughly how often; discourse analysis examines how something is said and what the saying accomplishes — what it presupposes, which cultural repertoires it draws on, and what identity it constructs for the writer. Thematic coding might tag a comment as "workload"; discourse analysis would ask whether the student positions the workload as the instructor's fault, their own responsibility, or an unfair imposition on a paying customer.

Does discourse analysis tell me how many students felt a certain way?

No. It is an interpretive method for understanding how accounts are constructed, not a counting method. It cannot produce a prevalence estimate, and reporting it as if it could is a misuse. If you need "what proportion of the cohort was dissatisfied," use a scale item or a thematic count; use discourse analysis to understand the meaning and framing behind those numbers.

What is an interpretative repertoire?

An interpretative repertoire is a recognisable cluster of terms, metaphors and figures of speech that people draw on to build a version of reality — for example, the "value for money" repertoire that frames a course as a purchased product. Spotting a repertoire in student feedback tells you the shared cultural framing students are using to make their judgement, which is often more revealing than the judgement itself.

Can this be automated with AI or sentiment analysis?

Not credibly. Sentiment analysis assigns a polarity; topic models cluster words; neither reads how an account positions its writer or what power relations it reproduces, which is the whole point of discourse analysis. AI tools are genuinely useful for the triage step — retrieving the focused sub-corpus a human analyst should study — but the analysis itself remains interpretive human work.

When should I use discourse analysis rather than a scalable method?

Use it on a purposive sample when you need to understand framing, assumptions or bias in a specific, important case — a difficult module, a contested redesign, a set of comments that surprised you, or a suspected pattern in how different instructors are described. It is too labour-intensive for the full institutional firehose; reserve it for questions of meaning where the how matters as much as the what.

How do I keep a discourse analysis rigorous rather than just cherry-picking quotes?

Antaki and colleagues (2003) catalogue the ways discourse "analysis" collapses into mere illustrated summary — over-quoting, taking sides, spotting isolated features. The guard is systematic, evidenced analysis: work through how each extract is constructed, consider alternative readings, ground every claim in the text, and present your interpretation as a defensible reading argued from the data rather than an obvious truth.

References

• Potter, J., & Wetherell, M. (1987). Discourse and Social Psychology: Beyond Attitudes and Behaviour. London: Sage.
• Wetherell, M. (1998). Positioning and interpretative repertoires: Conversation analysis and post-structuralism in dialogue. Discourse & Society, 9(3), 387–412. https://doi.org/10.1177/0957926598009003005
• Gee, J. P. (2014). An Introduction to Discourse Analysis: Theory and Method (4th ed.). London: Routledge.
• Rogers, R., Malancharuvil-Berkes, E., Mosley, M., Hui, D., & O'Garro Joseph, G. (2005). Critical discourse analysis in education: A review of the literature. Review of Educational Research, 75(3), 365–416. https://doi.org/10.3102/00346543075003365
• Antaki, C., Billig, M., Edwards, D., & Potter, J. (2003). Discourse analysis means doing analysis: A critique of six analytic shortcomings. Discourse Analysis Online, 1(1).

Related resources

• What Can Open-Text Student Comments Tell You That Likert Scores Cannot?
• Gendered Language in Student Comments: Why Men Are "Brilliant" and Women Are "Caring"
• The Framework Method for Open-Text Course Feedback: A Structured Alternative to Thematic Coding
• Grounded Theory for Open-Text Course Feedback: Building Explanation, Not Just Themes
• Narrative Inquiry for Course Evaluation: Reading Student Stories as Wholes, Not Fragments
• How Many Open-Text Comments Are Enough? Thematic Saturation in Course Evaluations