Is That Score Gap Real or Just Luck of the Draw? Permutation Tests for Course-Evaluation Comparisons

In brief

Before you announce that Instructor A "scored higher" than Instructor B, or that this year's cohort rated the course worse than last year's, ask whether the gap could easily have arisen by chance. The permutation test (also called a randomisation test) answers exactly that question with minimal assumptions: it repeatedly shuffles the response labels, recomputes the difference each time, and asks how often a difference as large as the observed one appears by pure reshuffling. If a gap that size shows up in 30% of random shuffles, it is noise; if it shows up in 1%, it is a signal. For the small, skewed, bounded Likert samples typical of course evaluation, this is often more defensible than a t-test — and it maps directly onto the intuition non-statisticians actually have about "could this just be luck?".

What the research says

The logic is old and exact. R. A. Fisher (1935), in The Design of Experiments, introduced the permutation argument through the famous "lady tasting tea" example: under the null hypothesis of no effect, every arrangement of the observed data is equally likely, so the exact probability of a result as extreme as the one observed can be computed by enumerating the arrangements. No appeal to a normal distribution is required; the reference distribution is generated from the data.

Michael Ernst (2004), Permutation Methods: A Basis for Exact Inference (Statistical Science, 19, 676–685), is the accessible modern review. His central points for a practitioner:

• Permutation methods are exact and distribution-free under a simple exchangeability assumption — that, under the null hypothesis, the observations could equally well have carried any of the group labels.
• What was infeasible in Fisher's era (enumerating all arrangements) is now trivial: with modern computing you either enumerate exactly for small samples or draw a large random sample of permutations (a Monte Carlo permutation test) to approximate the exact p-value to any desired precision.
• The approach generalises well beyond two-group comparisons to correlations, regression, and multi-group designs.

Permutation tests sit in the same modern-resampling family as the bootstrap, but answer a different question. As Ernst and standard texts (e.g. Good, 2005, Permutation, Parametric, and Bootstrap Tests of Hypotheses) put it: a permutation test is for hypothesis testing — is there a difference? — by reshuffling under a null of no effect; the bootstrap is for estimating uncertainty — how wide is the confidence interval? — by resampling with replacement. For course evaluation they are complements: use a permutation test to decide whether a gap is real, and the bootstrap (see our companion piece) to put an honest interval around each score.

Why it matters for course evaluation in practice

Course-evaluation comparisons routinely violate the assumptions of the tests applied to them:

• The data are not normal. Likert responses are bounded (1–5), discrete, and frequently ceiling-bunched near the top of the scale with a left skew. The classic t-test assumes approximately normal, equal-variance data; a permutation test on the mean difference (or, better, the median or a trimmed mean) makes no such assumption.
• Samples are small. A seminar of 12 students, a specialist elective, a single tutorial group — these are exactly the cases where the Central Limit Theorem does not rescue a t-test but a permutation test remains exact. This is the same small-n regime that motivates funnel plots and empirical-Bayes shrinkage for fair comparison.
• The question is genuinely comparative. "Did the redesign move the score?", "Is this section rating the course differently from that one?", "Is the gender gap in this instructor's ratings larger than chance?" — all are two-sample or correlational questions a permutation test answers directly.
• It is explainable. You can describe a permutation test to a non-statistician dean in one sentence — "we shuffled the responses thousands of times to see how often a gap this big appears by chance" — which matters when the output feeds a personnel or quality decision that must withstand challenge.

A worked sketch: Instructor A averages 4.3 over 15 responses, Instructor B 3.9 over 12. Pool all 27 responses, randomly split them into groups of 15 and 12, record the mean difference, repeat 10,000 times. If a difference of 0.4 or more occurs in, say, 22% of shuffles, the gap is well within chance — and no ranking, bonus, or "needs improvement" flag should rest on it.

Limitations and honest caveats

A rigorous reader should not treat permutation tests as a cure-all:

• Exchangeability is a real assumption. The exactness depends on observations being exchangeable under the null. If sections differ systematically in composition (prior ability, time of day, discipline), a raw permutation test can attribute a genuine confound to the instructor — exactly the Simpson's-paradox and confounding problems that no resampling method fixes by itself.
• It tests, it does not adjust. A significant permutation result says a difference is unlikely under the null; it does not tell you the difference is meaningful or fair. Effect size and practical significance still require judgement, and a tiny but "significant" gap on a huge cohort can be trivial.
• Multiplicity still bites. Running permutation tests across dozens of instructors or items inflates false positives just as any repeated testing does; correction (or a hierarchical model) is still needed.
• Ties and discreteness. Heavily tied Likert data produce a lumpy permutation distribution and a granular p-value; this is honest but can make borderline results unstable, and a sensible test statistic (e.g. a rank-based one) should be chosen deliberately.
• Not a fix for bias or non-response. Permutation inference assumes the responses you have are the units of analysis; it cannot correct for the self-selection and non-response biases that may have shaped who answered in the first place.

How Koji incorporates this

Koji is an AI-native course-evaluation platform, and its analysis and reporting layer is built to keep comparisons honest rather than to manufacture rankings:

• Chance-aware comparisons by default. When Koji's reporting compares two cohorts, sections, or time points, it is designed to express whether an observed gap is distinguishable from sampling noise using resampling-based inference appropriate to small, bounded Likert data — not a normal-theory test applied out of habit.
• Testing paired with estimation. Koji reports a difference alongside an uncertainty interval, reflecting the permutation-versus-bootstrap division of labour: the test says whether a gap is real, the interval says how precisely each score is known. This pairs naturally with the bootstrap intervals described in bootstrap confidence intervals for small classes.
• Plain-language output. Koji's reports are written for deans and QA officers, so a comparison is framed as "this gap is within what we would expect from chance" rather than a bare p-value, matching the explainability that makes permutation logic defensible in a personnel context.
• Guardrails against misuse. Bias-aware reporting is designed to flag when a comparison rests on very small samples or systematically different cohorts, so the exchangeability caveat is surfaced rather than hidden — and small-n instructor scores can be presented with shrinkage and funnel-plot context instead of a naked ranking.
• Beyond the number. Because a significant difference still needs a why, Koji's AI-moderated conversational interviews and automatic thematic analysis let you follow a real, chance-excluded gap into the open-text explanations behind it, rather than stopping at "significant".

Koji's core research platform at koji.so applies the same resampling-based, small-sample-honest analysis to product and customer research, where teams face the same temptation to over-read a two-point difference between small user segments.

Choosing a statistic and reading the output

A permutation test is only as sensible as the statistic you permute and the way you report the result. Three practical decisions matter:

• Pick a statistic that matches the question and the data. For skewed, ceiling-bunched Likert responses, the difference in means is legitimate but often less informative than the difference in medians, a trimmed mean, or a rank-based statistic such as the sum of ranks (which yields the permutation form of the Wilcoxon-Mann-Whitney test). Choose before you look at the data, not after, so the choice cannot be reverse-engineered to reach a desired p-value.
• Enumerate or simulate, deliberately. For very small samples you can enumerate every arrangement and get an exact p-value; for larger ones, draw many random permutations (10,000 is a common default) for a Monte Carlo approximation, and report the number of permutations used so the precision of the p-value is transparent.
• Report the effect, not just the verdict. A permutation p-value tells you whether a gap is bigger than chance; it does not tell you whether it matters. Always accompany it with the observed difference and, ideally, a bootstrap interval around each group's score, so a reader sees both "is it real?" and "how big and how precise?".

Finally, decide the family of comparisons in advance. If you will test many instructors or items, plan the multiplicity correction (or move to a hierarchical model) up front, so the permutation machinery is not quietly turned into a fishing expedition that manufactures a "significant" gap out of pure repetition.

Related resources

• Bootstrap confidence intervals for small classes
• Funnel plots for fair instructor comparison
• Empirical-Bayes shrinkage for evaluation scores
• Simpson's paradox in aggregating scores
• Should you adjust scores for class size and difficulty?
• Latent profile analysis for course-evaluation segments

References

• Fisher, R. A. (1935). The Design of Experiments. Oliver & Boyd, Edinburgh.
• Ernst, M. D. (2004). Permutation methods: A basis for exact inference. Statistical Science, 19(4), 676–685. https://doi.org/10.1214/088342304000000396
• Good, P. I. (2005). Permutation, Parametric, and Bootstrap Tests of Hypotheses (3rd ed.). Springer. https://doi.org/10.1007/b138696
• Ludbrook, J., & Dudley, H. (1998). Why permutation tests are superior to t and F tests in biomedical research. The American Statistician, 52(2), 127–132. https://doi.org/10.1080/00031305.1998.10480551
• Efron, B., & Tibshirani, R. J. (1993). An Introduction to the Bootstrap. Chapman & Hall.