Pass Rates Are Not the Objective Measure of Teaching Quality You Think They Are

Answer up front: When people lose faith in student satisfaction scores, the natural move is to reach for something that feels harder and more objective: pass rates, grade distributions, degree classifications — attainment. Surely how well students actually performed is a better measure of teaching quality than how they felt? It is not, at least not on its own. Attainment is confounded by who enrolled (intake ability), corrupted by grade inflation, shaped by how hard the assessment was set, and — worst of all — it gets worse as a quality signal the moment you make it a target, because the cheapest way to raise pass rates is to lower the bar. Attainment data belongs in a triangulated picture of teaching quality. It does not belong at the centre of one pretending to be objective.

The appeal of the hard number

The case against student ratings is well rehearsed, and much of it is fair: ratings can measure satisfaction rather than learning, they carry demographic biases, and they are vulnerable to leniency. Against that backdrop, attainment looks like solid ground. It is behavioural, not attitudinal. It sits in the student records system, audited and official. It maps onto outcomes everyone claims to care about. So committees reason: stop asking students how they felt and start looking at how they did.

The instinct is understandable. But it repeats the same error the ratings critics made — mistaking a number that is easy to obtain for a number that is valid. A pass rate is a measurement, and like any measurement it must be asked the awkward question: of what, exactly, and with what confounds?

Confound one: intake, not teaching

The most obvious problem is selection. A programme that admits the strongest-qualified applicants in the country will post high attainment almost regardless of how it is taught. A programme that widens access to students with lower prior attainment, weaker school backgrounds, or English as an additional language will post lower attainment even if its teaching adds far more value. Raw attainment measures the stock of ability a cohort walked in with at least as much as the value added while they were there — the same trap we described for reading graduate earnings as proof a course is good. To isolate teaching you would need a defensible value-added or learning-gain model, and even the UK's dedicated learning-gain pilots found that genuinely hard to pin down.

Confound two: grade inflation

If attainment were a stable yardstick, a first-class degree would mean the same thing across two decades. It does not. In England, the Office for Students found that the share of students awarded first-class honours rose from 15.5% in 2010–11 to 32.8% in 2021–22 — more than doubling — and that after accounting for observable factors such as prior entry qualifications and subject mix, around half of the 2021–22 figure (16.4 percentage points) was statistically unexplained by changes in the cohort (Office for Students, grade-inflation analysis). A yardstick that stretches by that much over a decade cannot anchor a claim about teaching quality. A rising pass rate might mean better teaching — or looser marking, softer assessment, or institutional pressure to award higher classifications.

Confound three: the assessment sets the score

Attainment is not an external fact about learning; it is produced by an assessment that someone in the department designed. Make the exam easier, weight the coursework more generously, or move the pass threshold, and attainment rises without a single student learning more. This is why attainment and teaching quality are entangled at the source: the same academics who teach the course often set and mark its assessment. Unlike an independent direct measure of learning — a validated concept inventory, an externally-benchmarked standard — an internal pass rate has no fixed reference point outside the programme that generated it.

Confound four — the fatal one: Goodhart's Law

Even if you could adjust away intake, inflation, and assessment difficulty, one problem is structural. The moment attainment becomes the official measure of teaching quality — tied to funding, rankings, or a lecturer's review — it stops being a measure and becomes a target. As we have written about Goodhart's Law in course evaluation, a measure under pressure distorts the behaviour it was meant to track. And the cheapest, fastest way to raise a pass rate is not better teaching; it is a lower bar. An attainment-centred quality regime quietly rewards exactly the grade inflation the sector says it wants to stop. Student ratings can be gamed by entertaining lectures and generous marking; attainment can be gamed even more directly, by marking generously alone.

The cross-border comparability trap

The problem multiplies the moment a comparison crosses a grading culture. Degree-classification conventions and marking norms differ sharply between national systems — and even between institutions within one country — so a first-class rate that looks strong in one setting may be unremarkable in another. A joint or transnational programme that ranks its partners on raw attainment stacks every confound above on top of a yardstick that is not even shared. If attainment cannot be compared across borders without adjusting for how grades are awarded, it cannot quietly anchor a quality judgement within a single institution either — the only difference is that the internal mismatch is harder to see, because everyone assumes a 2:1 means the same thing in every department. It does not.

But surely outcomes matter more than feelings?

Yes — and this is the honest counterargument, so let us take it seriously. It is genuinely true that a course where students demonstrably master the material is better than one where they leave satisfied but incompetent, and a quality system that ignored outcomes entirely would be indefensible. The argument here is not that attainment is irrelevant; it is that raw internal attainment is not a clean, objective read of teaching quality, because it confounds intake, inflation, and assessment design, and because it degrades under Goodhart pressure. The constructive version of "outcomes matter" is not the pass rate. It is a triangulated evidence base: value-added or learning-gain measures that adjust for intake; externally-referenced direct assessment against standards rather than internal grades alone; and, yes, student experience data to explain why the outcomes came out as they did. Outcomes matter so much that they deserve better evidence than an unadjusted pass rate.

Where course evaluation — and Koji — fit

None of this rehabilitates the naive satisfaction average; we have spent a great deal of this blog explaining its limits. The point is that no single number — satisfaction or attainment — is the objective measure of teaching quality, and the responsible move is triangulation: read attainment, direct learning evidence, and student experience together, each covering the others' blind spots.

This is where well-designed course evaluation earns its place, and where Koji for Education is built to help. Attainment tells you what happened; it is silent on why. Koji's AI-moderated conversational interviews probe the mechanism behind the numbers — whether a low pass rate reflects weak teaching, an over-hard assessment, or a struggling cohort, and whether a high pass rate reflects genuine mastery or a course everyone found undemanding. Its automatic thematic analysis surfaces those explanations at programme scale, and its programme- and institution-level reporting lets you set student-experience evidence beside attainment rather than choosing between them. Koji does not claim to measure learning directly — no feedback survey does — but it supplies the interpretive layer that turns a bare pass rate into something a quality committee can actually reason about. (The same conversational engine underpins the wider research platform at koji.so.)

Reaching for pass rates because ratings feel too soft is understandable. But it swaps one flawed single number for another that is inflated, confounded, and easier to game. There is no objective measure of teaching quality waiting in the student records system. There is only better and worse evidence, read honestly and together.

Closing the loop

If you want the why behind your attainment data — the explanation a grade distribution can never give — see how Koji for Education gathers programme-level student evidence. Triangulate. Do not substitute one flawed number for another.