What Will This Instructor Score Next Term — With a Guarantee? Conformal Prediction for Course Evaluation

In brief

Conformal prediction is a distribution-free method that turns any predictive model into one that outputs an interval (or a set) guaranteed to contain the true value at a coverage rate you specify — say 90% — without assuming the data are normal, without large samples, and without trusting that your model is correctly specified. For course evaluation, it answers a question raw averages and even confidence intervals dodge: given this instructor is history and this cohort, what range will next term is score plausibly fall in, and how often will that promise be kept? The guarantee is marginal and finite-sample, which is exactly what a fairness-minded quality office wants when a single number is about to inform a personnel or programme decision.

What the research says

Conformal prediction was developed by Vladimir Vovk, Alexander Gammerman and Glenn Shafer and set out in full in their 2005 monograph Algorithmic Learning in a Random World. The accessible entry point is Shafer and Vovk is 2008 tutorial in the Journal of Machine Learning Research (9:371-421), which states the core promise plainly: if successive examples are drawn independently from the same distribution (more precisely, are exchangeable), then a conformal predictor built at confidence level 1 minus epsilon will be wrong at most an epsilon fraction of the time, even though it learns from an accumulating dataset rather than from independent ones. The method wraps around any underlying algorithm — a mean, a ridge regression, a random forest — and converts its raw output into a valid interval.

The mechanism is a ranking argument, not a distributional one. In the common split (inductive) version, you divide data into a training part and a calibration part. You fit any model on the training part, then compute a nonconformity score for every calibration point — typically the absolute residual, how far the true value sat from the model is prediction. To predict a new case, you ask: for which candidate values would this new point is nonconformity score not be unusually large relative to the calibration scores? The interval is the set of candidate values that survive. Because it is built from the empirical rank of residuals, its coverage holds for any underlying distribution — hence distribution-free.

Anastasios Angelopoulos and Stephen Bates is 2023 monograph A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification (Foundations and Trends in Machine Learning 16(4):494-591, doi:10.1561/2200000101) made the split-conformal recipe a practical default: with a calibration set of size n, take the ceiling of (n+1)(1 minus epsilon) / n empirical quantile of the calibration scores, and the resulting interval has coverage of at least 1 minus epsilon in finite samples. Jing Lei and colleagues (2018, Journal of the American Statistical Association 113(523):1094-1111, doi:10.1080/01621459.2017.1307116) gave the regression theory, including conformalized residual and locally-weighted variants that let interval width adapt to how noisy each case is — wider intervals for small, volatile classes and narrower ones for large, stable ones.

Why it matters for course evaluation in practice

Quality offices routinely act on point estimates: a 4.1 this year, a projected 4.3 after an intervention, a flag when a score dips. Every honest reader knows those numbers carry error, but the tools normally used to express it come with baggage. A textbook confidence interval assumes a sampling model and describes the mean, not a future single value. A bootstrap interval relaxes the normality assumption but still targets a parameter and can undercover badly in the tiny classes where decisions are most contested. Conformal prediction targets the thing a committee actually cares about — the next realised value — and its coverage guarantee does not depend on the model being right.

Three concrete uses stand out. First, honest forecasting: if you predict an instructor is next-term score from their history and course features, a 90% conformal interval tells the committee how much the prediction can be trusted; a width of plus or minus 0.6 on a five-point scale is itself the finding. Second, defensible flagging: instead of flagging any score below a fixed cut, flag only cases whose conformal interval sits entirely below the standard, which controls false alarms in a way that pairs naturally with funnel-plot logic. Third, change detection: if the observed score falls outside the interval predicted from prior terms, that is genuine evidence something moved — a cleaner test than comparing two noisy means and falling into the 4.2-vs-4.4 trap.

Limitations and honest caveats

Conformal prediction is not magic, and a PhD reader will press on four points. Exchangeability is the real assumption. The guarantee holds if new cases are exchangeable with calibration cases; it fails under drift. A curriculum reform, a modality shift, or a grading-policy change breaks exchangeability, and coverage silently degrades — the very moments you most want a forecast are the moments the assumption is weakest. Coverage is marginal, not conditional. A 90% predictor is right 90% of the time on average across all cases, not necessarily 90% within each subgroup; small departments, or instructors teaching atypical courses, can be systematically under-covered even while the global rate is honoured. Mondrian and locally-weighted conformal variants mitigate this but do not fully solve it. Intervals can be wide and uninformative. Distribution-free validity is bought with width; if the underlying signal is weak, an honest conformal interval may span most of the scale, which is a true statement but a frustrating one. It quantifies predictive uncertainty, not bias. If your evaluation instrument is contaminated by the biases documented elsewhere in this knowledge base, conformal intervals will faithfully cover a biased target. Validity of coverage is not validity of the measure.

How Koji incorporates this

Koji is analytics layer is built to report ranges and decision-relevant uncertainty rather than bare point estimates, and the conformal logic maps onto that directly. When Koji projects a course or instructor result, it is designed to attach a distribution-free interval calibrated on the institution is own prior cohorts, so the number a committee sees is paired with an explicit, guarantee-backed range rather than a false-precision decimal. Because the method wraps any underlying model, it composes with the shrinkage and multilevel estimates Koji already produces — the point estimate can be an empirical-Bayes shrunk score while the interval around it is conformal. Koji is review-flagging is designed to trigger on whole-interval evidence (the interval clearing a threshold) rather than a raw mean crossing a line, which pairs with its type-S/type-M and misclassification safeguards to keep small-cohort decisions honest. And because exchangeability is the load-bearing assumption, Koji surfaces the context a reviewer needs to judge it — cohort size, modality changes, response-rate shifts — rather than hiding a forecast behind a single confident-looking figure. Koji is core research platform at koji.so applies the same AI-moderated interview engine and uncertainty-aware reporting to product and customer research, where distribution-free intervals on predicted metrics are just as useful.

The honest framing for a quality office is this: conformal prediction does not make a noisy evaluation precise. It makes the imprecision legible and bounded, so a committee can see when a projected score is trustworthy enough to act on and when the only defensible reading is we do not yet know.

Frequently asked questions

Is a conformal interval the same as a confidence interval?

No. A confidence interval is a statement about a parameter (usually a mean) under a sampling model. A conformal interval is a statement about a future observation and holds distribution-free. They answer different questions; for deciding what a single instructor might realistically score next term, the prediction interval is the relevant object.

How much data do I need to calibrate one?

Split conformal works with modest calibration sets — even a few dozen prior course-terms give a usable, if wide, interval, and the finite-sample coverage guarantee holds exactly at those sizes. More calibration data mainly buys narrower intervals, not valid ones; validity is there from the start.

What breaks the guarantee?

Departures from exchangeability. If next term is systematically different from your calibration history — a new delivery mode, a reformed curriculum, a changed rating scale — coverage is no longer guaranteed. Treat a structural change as a reason to recalibrate, not to trust an old interval.

Does conformal prediction fix bias in course evaluations?

No. It controls the coverage of an interval around whatever target your model predicts. If that target is a biased score, the interval will reliably cover the biased value. Use it alongside validity and bias analyses, not as a substitute for them.

Can it produce a set of categories instead of a numeric range?

Yes. For classification-style outputs (for example, predicting a rating band or a theme label), conformal prediction returns a set of plausible labels guaranteed to contain the truth at the chosen rate, which is often more honest than forcing a single label.

Why choose it over just reporting the observed average?

The observed average tells you what happened; it says nothing calibrated about what will happen or how much a projection can be trusted. Conformal prediction adds a guarantee-backed range to any forecast, which is what a committee needs before acting on a number.

References

• Vovk, V., Gammerman, A., & Shafer, G. (2005). Algorithmic Learning in a Random World. Springer. doi:10.1007/b106715
• Shafer, G., & Vovk, V. (2008). A tutorial on conformal prediction. Journal of Machine Learning Research, 9, 371-421.
• Lei, J., G Sell, M., Rinaldo, A., Tibshirani, R. J., & Wasserman, L. (2018). Distribution-free predictive inference for regression. Journal of the American Statistical Association, 113(523), 1094-1111. doi:10.1080/01621459.2017.1307116
• Angelopoulos, A. N., & Bates, S. (2023). A gentle introduction to conformal prediction and distribution-free uncertainty quantification. Foundations and Trends in Machine Learning, 16(4), 494-591. doi:10.1561/2200000101

Related Resources

• Fair Confidence Intervals for Small Classes: The Bootstrap for Course-Evaluation Reporting
• The 4.2 vs 4.4 Trap: Why Small Differences in Course-Evaluation Means Are Usually Noise
• Stop Comparing Raw Averages: Empirical-Bayes Shrinkage for Course-Evaluation Scores
• Which Instructor Scores Are Genuinely Unusual? Funnel Plots for Fair Course-Evaluation Comparison
• Your 'Significant' Instructor Difference Might Point the Wrong Way: Type S and Type M Errors
• Even Fair Evaluations Can Rank Unfairly: Why Course-Evaluation Scores Misclassify Instructors