You Flagged the Lowest Instructor, Then Tested If They Were Below Average: Selective Inference and the Winner's Curse

In brief

If you scan sixty instructors, flag the one with the lowest average, and then run a test asking "is this instructor significantly below the department mean?", the p-value and confidence interval you get are not valid. The act of selecting the most extreme case before testing inflates significance and biases the flagged estimate away from the truth — a phenomenon known variously as the winner's (or loser's) curse, selection bias, and the problem of post-selection inference. The fix is not a better threshold but a different kind of inference: selection-adjusted intervals, simultaneous validity across all comparisons you could have made, or empirical-Bayes shrinkage of the flagged score before you act on it.

What the research says

Classical confidence intervals and hypothesis tests assume the quantity being tested was chosen before the data were seen. Course-evaluation dashboards routinely violate this: they compute every instructor's mean, sort them, and invite the reader to test whichever one now looks worst. Berk, Brown, Buja, Zhang and Zhao (2013) formalised the consequence in "Valid Post-Selection Inference" (PoSI). They show that when the target of inference is chosen by looking at the data, the nominal coverage of a standard 95% interval can fall well short of 95%, and they construct intervals that remain valid simultaneously over every model or contrast the analyst might have selected. That universal validity is bought with width: PoSI intervals are wider than naive ones, and the penalty grows with the number of candidate comparisons.

Taylor and Tibshirani (2015), writing in PNAS, frame the same issue as selective inference: rather than protecting against every possible selection, condition the inference on the specific selection event that actually occurred (for example, "this instructor was the minimum of sixty"). Conditioning on selection yields tests and intervals whose error rates are correct given the way the case was chosen, and the approach extends cleanly to forward-stepwise selection, the lasso, and other data-driven procedures.

A third, complementary tradition treats the problem as one of biased point estimation. Benjamini and Yekutieli (2005) introduced the false coverage-statement rate (FCR): when you build intervals only for a data-selected subset of parameters, the expected proportion of those intervals that miss their target can exceed the nominal level. Their remedy constructs each selected interval at a stricter confidence level — roughly 1 − q·R/m, where R of m candidates were selected — restoring guaranteed coverage across the reported set. Efron (2011), in "Tweedie's Formula and Selection Bias," shows that the largest (or smallest) few of many estimates systematically overstate their true values, and that an empirical-Bayes correction — shrinking each selected estimate toward the overall distribution using the marginal density of the data — undoes much of the bias. This is the estimation-side face of what regression to the mean describes descriptively.

Why it matters for course evaluation in practice

The practical trap is a two-step ritual almost every quality office performs: rank, then test. A department runs sixty sections, the automated report highlights the lowest-scoring instructor, and a committee asks whether that score is "really" below average. Because the instructor was chosen for being extreme, three things are simultaneously true and easy to miss. First, the observed score is a biased estimate — the minimum of many noisy means is expected to sit below the true value even if that instructor is merely average and unlucky. Second, the naive test overstates significance: you have implicitly run sixty comparisons and reported the most extreme, so a "p < 0.05" against a fixed benchmark is not a 5% false-alarm rate. Third, next term the flagged instructor will very likely "improve" toward the mean with no intervention at all, which can be misread as evidence that a warning worked.

Selection-aware reasoning changes the decision. If the department screens sixty instructors, an FCR-adjusted or simultaneous interval for the flagged case is materially wider than the naive one, and often it will straddle the department mean — telling you the flag is not yet actionable evidence. An empirical-Bayes-shrunk estimate pulls the alarming raw score back toward the distribution, giving a fairer "best guess" of the instructor's true standing on which to base a conversation rather than a sanction. The same logic protects the top of the table: the highest-rated instructor is, by identical reasoning, probably not as far ahead as the raw number claims, which matters for teaching awards and promotion cases.

Limitations and honest caveats

Selective-inference methods are not a free pass. Their central cost is conservativeness: PoSI intervals in particular widen with the number of candidate comparisons, and if a department screens hundreds of items the guaranteed-valid interval can become too wide to be useful. Conditional selective inference is sharper but requires the selection rule to be explicit and known — "I tested the minimum of sixty" is analysable; "these three caught my eye while I was reading" is not, and no procedure fully rescues inference after informal, undocumented selection. Empirical-Bayes shrinkage assumes there are enough units (instructors, sections) to estimate the prior distribution reasonably, and it can under-correct genuine outliers, nudging a truly poor performer toward the mean along with the merely unlucky. Finally, all of these tools address statistical selection bias; they cannot repair a biased instrument. If the underlying ratings carry the gender, discipline, or workload confounds documented elsewhere in this knowledge base, correcting the selection does nothing about the confound. The honest framing is that selection adjustment is necessary, not sufficient — it stops you over-reading noise, but the substantive validity of the score is a separate question.

How Koji incorporates this

Koji is designed to make selection-aware reporting the default rather than an afterthought. Instructor and section summaries are presented with empirical-Bayes-shrunk estimates alongside the raw mean, so a flagged low score is shown next to the fairer, distribution-informed best guess rather than as a bare extreme. Comparison views are built to reflect how many units were screened, so an interval for "the lowest of sixty" is not silently reported as though that instructor had been the only one examined. Where a flag is meant to trigger action, Koji's workflow encourages a flag-then-confirm pattern: the case identified in one collection window is re-examined against a fresh window of data — a form of data splitting — so the confirmatory read is not contaminated by the selection that produced the flag. And because a single number is never enough for a personnel decision, Koji's AI-moderated conversational interviews and automatic thematic analysis let a committee triangulate a flagged score against what students actually wrote, so the human judgement rests on evidence, not on the winner's curse. The same machinery underpins Koji's core research platform at koji.so, where teams running product and customer research face the identical trap — flagging the "worst" feature or the "best" segment from a survey and then testing it on the data that selected it.

Frequently asked questions

Is this the same as the multiple-comparisons problem?

It is closely related but not identical. Multiple-comparison corrections such as the false discovery rate protect a fixed, pre-specified family of tests. Selective inference protects against a data-driven choice of which parameter to test or report — you did not decide in advance to test this instructor; the data decided for you. The two often appear together, and the FCR framework is precisely the bridge between them.

How is the winner's curse different from regression to the mean?

They describe the same underlying fact from different angles. Regression to the mean is the descriptive observation that an extreme score tends to be less extreme on re-measurement. The winner's curse is the inferential consequence: the selected extreme estimate is a biased estimator of the true value, and Efron's shrinkage is the correction. Our companion note on regression to the mean covers the year-over-year pattern; this article covers the inference you draw at the moment of flagging.

Does this only matter for the lowest scores?

No. Selection bias is symmetric. The highest-rated instructor selected from a large pool is also over-estimated, which matters for teaching prizes, merit pay, and promotion evidence. Any time you pick an extreme because it is extreme and then report an estimate or test for it, the adjustment applies.

What is the simplest defensible fix for a small quality office?

Two low-cost moves go a long way. First, shrink: report an empirical-Bayes-adjusted score next to the raw one, which our empirical-Bayes shrinkage note explains. Second, split: confirm any flag on a second window of data the flag did not use. Neither requires exotic software.

Can I just widen my confidence intervals a bit?

Widening is the right instinct, but by how much is the technical question. The FCR procedure gives a principled amount — tightening the per-interval level in proportion to how many candidates you screened. An arbitrary widening neither guarantees coverage nor is defensible to a critical reader.

Should we stop flagging outliers altogether?

No — flagging is a reasonable way to allocate attention. The point is that a flag is a hypothesis-generating signal, not a hypothesis-confirming one. Use it to decide where to look more closely, then gather selection-independent evidence before you act.

References

• Berk, R., Brown, L., Buja, A., Zhang, K., & Zhao, L. (2013). Valid Post-Selection Inference. Annals of Statistics, 41(2), 802–837. https://doi.org/10.1214/12-AOS1077
• Taylor, J., & Tibshirani, R. J. (2015). Statistical learning and selective inference. Proceedings of the National Academy of Sciences, 112(25), 7629–7634. https://doi.org/10.1073/pnas.1507583112
• Benjamini, Y., & Yekutieli, D. (2005). False Discovery Rate–Adjusted Multiple Confidence Intervals for Selected Parameters. Journal of the American Statistical Association, 100(469), 71–81. https://doi.org/10.1198/016214504000001907
• Efron, B. (2011). Tweedie's Formula and Selection Bias. Journal of the American Statistical Association, 106(496), 1602–1614. https://doi.org/10.1198/jasa.2011.tm11181

Related resources

• Compare 60 Instructors, Expect 3 False Alarms: Multiple Comparisons and the False Discovery Rate
• Stop Comparing Raw Averages: Empirical-Bayes Shrinkage for Course-Evaluation Scores
• Why a Low-Scoring Course Usually 'Improves' Next Year: Regression to the Mean
• Your 'Significant' Instructor Difference Might Point the Wrong Way: Type S and Type M Errors
• Which Instructor Scores Are Genuinely Unusual? Funnel Plots for Fair Comparison
• Where Should You Set the Review Trigger? Signal Detection Theory for Course-Evaluation Thresholds