Compare 60 Instructors, Expect 3 False Alarms: Multiple Comparisons and the False Discovery Rate

In short: If you test each of 60 instructors against the department average at the usual 5% significance level and every instructor is genuinely identical, you should still expect about three to be flagged as "significantly different" by chance alone. This is the multiple-comparisons problem. Two families of fixes exist: controlling the family-wise error rate (the Bonferroni correction makes any false flag unlikely but misses real problems), and controlling the false discovery rate (Benjamini & Hochberg, 1995), which caps the proportion of your flags that are false and is usually the right trade-off for screening many instructors. Neither is a substitute for shrinkage and effect-size thinking, but ignoring multiplicity guarantees you will chase noise.

The problem: every extra test buys another chance to be fooled

Quality-assurance work is, statistically, a machine for running many tests at once. A dashboard compares every instructor to the faculty mean. A programme review checks each of twenty items for a "significant" year-on-year change. A dean asks which modules fell below threshold this semester. Each of these is not one hypothesis test but dozens or hundreds run in parallel — and the arithmetic of parallel testing is unforgiving.

A single test at significance level α = 0.05 has a 5% chance of a false positive when nothing is really going on. Run the test independently on 60 instructors who are all genuinely average, and the probability that at least one is flagged is 1 − (0.95)^60 ≈ 0.95. You are almost certain to "find" a significant outlier. The expected number of false flags is 60 × 0.05 = 3. Scale to a university comparing 500 course sections and you should expect roughly 25 spurious "significant" results every cycle, purely from sampling noise. Act on those, and you will be counselling instructors, redesigning courses, and writing accreditation narratives about differences that do not exist.

What the research says

The multiple-comparisons problem is old and well-formalised, and the literature offers two distinct goals you can control.

Family-wise error rate (FWER) is the probability of making even one false flag across the whole family of tests. The classic tool is the Bonferroni correction: test each hypothesis at α/m instead of α, where m is the number of tests. For 60 instructors, that means judging each at 0.05/60 ≈ 0.0008. Dunn (1961), in the Journal of the American Statistical Association, gave the rigorous treatment of this approach for comparisons among means. Bonferroni is simple and makes any false alarm very unlikely — but that protection is bought at a steep price in power. Corrected to 0.0008, you will miss all but the most extreme genuine problems, which is a poor bargain when the whole point of screening is to catch issues early.

The decisive advance was Benjamini and Hochberg (1995), "Controlling the false discovery rate: A practical and powerful approach to multiple testing," in the Journal of the Royal Statistical Society, Series B. Instead of guarding against any false flag, they proposed controlling the false discovery rate (FDR) — the expected proportion of your flags that are false. If you set FDR at 10%, then among the instructors you flag, on average no more than one in ten is a false alarm; the rest are real. The Benjamini–Hochberg procedure is mechanical: rank the m p-values from smallest to largest, find the largest rank k for which p(k) ≤ (k/m) × q (where q is your chosen FDR), and flag everything up to that rank. Because it adapts to how many genuine signals are present, it is far more powerful than Bonferroni while still bounding the error you care about in a screening context — the fraction of your action list that is noise.

Benjamini and Yekutieli (2001), in the Annals of Statistics, extended FDR control to correlated tests — important because instructors within a department, or items within a survey, are not independent. Under general dependence, the procedure still controls FDR after a modest logarithmic adjustment to q.

There is a genuine counter-current worth taking seriously. Gelman, Hill and Yajima (2012), writing in the education-focused Journal of Research on Educational Effectiveness, argued that we "usually don''t have to worry about multiple comparisons" if we abandon the classify-each-test framework altogether and instead fit a single multilevel (hierarchical) model that partially pools estimates toward the group mean. Shrinkage does much of the work a multiplicity correction does — it pulls extreme small-sample estimates back toward average — without the awkward machinery of counting tests. This is the same logic behind empirical-Bayes reporting of instructor scores. Their point is not that multiplicity is harmless but that the honest fix is often a better model, not a p-value penalty bolted onto a bad one.

Why it matters for course evaluation in practice

The multiplicity trap shows up wherever course-evaluation data is scanned for outliers:

• Instructor league tables. Ranking every instructor and flagging those "significantly" above or below the mean runs one test per instructor. Without a multiplicity control, the bottom of the table will always contain names that are there by luck, and those are exactly the people who get scrutinised.
• Year-on-year "significant change" alerts. A dashboard that highlights every item that moved significantly since last year is running one test per item per unit. Most highlighted moves in a large system are noise — and, compounded with regression to the mean, actively misleading.
• Subgroup breakdowns. Slicing a course''s ratings by gender, year of study, domestic/international status, and delivery mode multiplies the tests fast. "The course scored significantly lower among second-year international students" is the kind of claim that appears somewhere in almost any dataset if you cut it enough ways.
• Accreditation evidence. Presenting a handful of "statistically significant" findings cherry-picked from a large battery, without disclosing how many comparisons were made, is a reviewer red flag and, increasingly, an ethics one.

The practical upshot: decide before you look how you will control error across the family of comparisons, report how many tests were run, and prefer FDR control (or a hierarchical model) over both naive testing and over-conservative Bonferroni when the task is to produce a credible short-list for follow-up.

Limitations and honest caveats

• Statistical significance is not the right target anyway. Even a perfectly multiplicity-corrected "significant" gap between a 4.2 and a 4.3 can be trivially small. Multiplicity control answers "is this gap distinguishable from noise?"; it does not answer "is this gap large enough to matter?" You still need effect sizes and confidence intervals.
• The choice of q (or α) is a value judgement. Setting FDR at 5%, 10%, or 20% trades missed real problems against wasted follow-up on false ones. There is no statistically "correct" level — it depends on the cost of each error in your context, which is a governance decision, not a computation.
• FDR controls an average, not a guarantee. Controlling FDR at 10% does not promise that this particular flagged list is 10% false; it bounds the expected proportion across many such analyses. In a single small analysis the realised proportion can be higher.
• Dependence and sparse data complicate everything. With few respondents per class, p-values themselves are unstable, and the independence assumptions behind the simplest procedures fail. Benjamini–Yekutieli helps, but the cleaner answer in small-sample, nested settings is often the multilevel model Gelman and colleagues advocate rather than a correction layered on fragile individual tests.
• A correction cannot rescue a biased comparison. If the underlying scores are contaminated by response-rate differences or demographic confounds, controlling multiplicity just gives you a statistically disciplined route to the wrong conclusion.

How Koji incorporates this

Koji for Education is built so that "who looks unusual?" is answered with multiplicity and shrinkage in mind, not with a wall of naive significance stars:

• Shrinkage-first reporting. Rather than testing every instructor against the mean independently, Koji''s reporting leans on partial pooling — small classes are pulled toward the group average unless the evidence is strong — which is the same hierarchical logic Gelman, Hill and Yajima recommend as the principled alternative to counting tests. This is described in Koji''s empirical-Bayes shrinkage and multilevel-models guidance.
• Visual screening that encodes expected variation. Koji supports funnel-plot-style comparison, where control limits widen for small classes so that only genuinely unusual scores fall outside — a graphical multiplicity discipline that resists the "someone is always at the bottom" trap.
• Effect sizes and intervals, not just flags. Because a multiplicity-corrected flag can still be trivial, Koji pairs any comparison with the magnitude of the difference and its uncertainty, consistent with its small-mean-differences and misclassification guidance — so a "significant" result is never presented as an actionable one on its own.
• Transparent scope of comparison. When Koji surfaces outliers or subgroup differences, the analysis is framed around a pre-specified question rather than an unlimited fishing expedition across every possible slice, which is the practical way to keep the false-discovery rate honest.

Koji is designed to reduce the chance that a QA office acts on noise; it does not promise that any single flagged list is free of false positives — the statistics themselves forbid that guarantee. The same discipline underpins the AI-moderated research on Koji''s core platform at koji.so, where scanning many segments for "significant" differences carries exactly the same multiplicity risk.

Frequently asked questions

Why do I get "significant" outliers even when all my instructors are equally good? Because each significance test has a built-in false-positive rate. At the standard 5% level, testing 60 identical instructors yields about three flags by chance (60 × 0.05), and at least one flag is nearly certain. The outliers are an artefact of running many tests, not evidence that those instructors differ.

What is the difference between the Bonferroni correction and the false discovery rate? Bonferroni controls the family-wise error rate — the chance of even one false flag — by dividing your significance level by the number of tests. It is very strict and misses most real problems. The false discovery rate (Benjamini & Hochberg, 1995) controls the proportion of your flags that are false, which is usually what a screening exercise cares about, and it retains far more power to detect genuine issues.

Which correction should a QA office use for instructor comparisons? For screening many instructors to build a follow-up short-list, FDR control (Benjamini–Hochberg) is usually the best trade-off. If a single false accusation carries serious consequences — a personnel decision — the stricter family-wise control is more defensible. Often the cleanest approach is neither: fit a multilevel model that shrinks small-sample estimates toward the mean, as Gelman, Hill and Yajima (2012) recommend.

Does controlling the false discovery rate guarantee my flagged list is 90% correct? No. Setting FDR at 10% bounds the expected proportion of false flags averaged over many analyses; the realised proportion in one particular list can be higher, especially with few tests. It is a discipline that keeps false discoveries bounded on average, not a certificate for a single report.

Do I still need effect sizes if I control for multiple comparisons? Yes, and this is essential. Multiplicity control only tells you whether a difference is distinguishable from noise, not whether it is large enough to matter. A multiplicity-corrected "significant" gap of 0.1 on a five-point scale is still practically meaningless, so always pair any flag with the size of the difference and its confidence interval.

How does this relate to regression to the mean? They compound. The instructors flagged as extreme this cycle are disproportionately there by luck, so many will "improve" or "decline" next cycle purely through regression to the mean — and a naive year-on-year significance scan will then flag that movement too. Controlling multiplicity and using shrinkage together blunt both artefacts.

Related resources

• Which Instructor Scores Are Genuinely Unusual? Funnel Plots for Fair Course-Evaluation Comparison
• Stop Comparing Raw Averages: Empirical-Bayes Shrinkage for Course-Evaluation Scores
• The 4.2 vs 4.4 Trap: Why Small Differences in Course-Evaluation Means Are Usually Noise
• Is That Score Gap Real or Just Luck of the Draw? Permutation Tests for Course-Evaluation Comparisons
• Even Fair Evaluations Can Rank Unfairly: Why Course-Evaluation Scores Misclassify Instructors
• Students Are Nested in Courses: Why Multilevel Models Beat Raw Averages for Course Evaluation

References

• Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B (Methodological), 57(1), 289–300. https://doi.org/10.1111/j.2517-6161.1995.tb02031.x
• Benjamini, Y., & Yekutieli, D. (2001). The control of the false discovery rate in multiple testing under dependency. The Annals of Statistics, 29(4), 1165–1188. https://doi.org/10.1214/aos/1013699998
• Dunn, O. J. (1961). Multiple comparisons among means. Journal of the American Statistical Association, 56(293), 52–64. https://doi.org/10.1080/01621459.1961.10482090
• Gelman, A., Hill, J., & Yajima, M. (2012). Why we (usually) don''t have to worry about multiple comparisons. Journal of Research on Educational Effectiveness, 5(2), 189–211. https://doi.org/10.1080/19345747.2011.618213