Fair Confidence Intervals for Small Classes: The Bootstrap for Course-Evaluation Reporting

In brief

When a course has a handful of respondents and a lopsided rating distribution — the norm, not the exception, in higher education — the standard "mean ± 1.96 × standard error" confidence interval is untrustworthy, and the bootstrap offers a more honest alternative by resampling the observed responses thousands of times to see how much the statistic actually wobbles. Introduced by Efron (1979) and developed in Efron and Tibshirani's (1993) textbook, the bootstrap makes no assumption that ratings are normally distributed. Efron and DiCiccio (1996) refined the interval construction (the bias-corrected and accelerated, or BCa, method) to handle the skew that course-evaluation data almost always shows. The bootstrap is not magic: with very small samples it can be unstable and it cannot conjure information that a handful of biased respondents never contained. But it is a substantial improvement over pretending a 7-response mean is precise.

What the research says

The classical confidence interval for a mean relies on the Central Limit Theorem: with a large enough sample, the sampling distribution of the mean is approximately normal, so mean ± 1.96 standard errors covers the true value about 95% of the time. Course-evaluation data violate the spirit of this in two ways at once. First, samples are often small — a seminar of 12 with 7 respondents. Second, the ratings are bounded and skewed — a 1–5 scale piled up near 4–5 with a ceiling, so the distribution is nothing like normal. Under these conditions the normal-theory interval can be too narrow, mis-centred, or spill past the ends of the scale, giving a false sense of precision.

Bradley Efron's (1979) paper in The Annals of Statistics, "Bootstrap methods: another look at the jackknife," introduced a different idea: instead of assuming a distribution, use the data as its own population. Draw many new samples of the same size with replacement from the observed responses, compute the statistic (mean, median, top-box percentage) on each, and use the spread of those thousands of recomputed statistics as a direct, empirical estimate of the sampling variability. The percentile bootstrap then reads a 95% interval straight off the 2.5th and 97.5th percentiles of that bootstrap distribution — no normality assumption required.

Efron and Tibshirani's (1993) An Introduction to the Bootstrap (Chapman & Hall) is the standard reference and shows the method working across means, medians, ratios and regression coefficients. Because the naive percentile interval can be biased or mis-centred when the underlying statistic is skewed — exactly the course-evaluation case — Efron and DiCiccio (1996), in Statistical Science, surveyed improved procedures. Their bias-corrected and accelerated (BCa) interval adjusts for both bias and skewness in the bootstrap distribution and achieves more accurate coverage (second-order accuracy) than the plain percentile method, while the bootstrap-t and ABC methods offer related refinements. For applied reporting on skewed ordinal-ish rating data, BCa is usually the sensible default.

The evidence base here is methodological rather than about teaching per se: it establishes that resampling gives valid uncertainty estimates under weak assumptions, and that skew-aware variants (BCa) correct the shortcomings of the simplest version. That is precisely the toolkit small-class evaluation reporting needs.

Why it matters for course evaluation in practice

The single most common reporting sin in course evaluation is presenting a mean from a tiny, self-selected sample as if it were a precise number — "4.1 this year versus 4.4 last year, a decline." With seven respondents and a ceiling-bound scale, the uncertainty around each of those means is wide enough that the "decline" may be indistinguishable from noise. A bootstrap confidence interval makes that uncertainty visible: reported as "4.1 (95% CI 3.4–4.6)," the number stops masquerading as a point estimate and starts communicating how little the class actually told us.

This matters most exactly where decisions are riskiest: small classes, personnel comparisons, and year-over-year change. Ranking instructors or flagging a module on a bare mean, when the intervals overlap heavily, is a documented misclassification hazard. Bootstrap intervals are also the honest way to put uncertainty on statistics for which no neat closed-form standard error exists — the median, the top-box percentage ("% who rated 4 or 5"), or a Net-Promoter-style figure — because you resample and recompute whatever statistic you report. And because the method is distribution-free, it handles the skew and ceiling effects that make normal-theory intervals for these scales unreliable.

Used well, the bootstrap turns a course-evaluation report from a league table of false precision into a set of ranges that support defensible conclusions: "we cannot distinguish these two sections," or "this drop is larger than sampling noise would produce."

Limitations and honest caveats

The bootstrap does not fix bias — only quantifies variance. This is the most important caveat. If the seven respondents are the seven angriest (or happiest) students, the bootstrap will faithfully report a tight interval around a biased mean. Resampling reflects the data you have; it cannot correct non-response bias, self-selection, or an unrepresentative sample. A narrow bootstrap CI on a 30%-response-rate evaluation is precise about the wrong quantity.

Very small n is genuinely hard. With n = 5–8 there are only a few distinct observed values, so the bootstrap distribution is coarse and its tails are poorly estimated; percentile and even BCa intervals can under-cover. The bootstrap mitigates small-sample problems but does not abolish them — below a threshold, the honest report is "too few responses to estimate reliably," and shrinkage/empirical-Bayes methods that borrow strength across courses may serve better.

Method and implementation choices matter. Results depend on the number of resamples (use several thousand, not a few hundred), the statistic chosen, and the interval variant; the naive percentile method can be noticeably off for skewed data, which is why BCa exists. Bootstrapping ordinal Likert data as if it were continuous inherits all the usual ordinal-vs-interval objections. And the bootstrap assumes observations are independent — cluster or repeated-measures structure needs a matched resampling scheme.

It is not a significance-test loophole. Overlapping bootstrap intervals are informative, but reading them as formal hypothesis tests, or fishing across many courses for non-overlaps, reintroduces multiple-comparison problems the interval alone does not solve.

In short: the bootstrap is the right tool for representing uncertainty under realistic distributions, and the wrong tool if it is used to lend spurious authority to a small, biased sample.

How Koji incorporates this

Koji is designed to report evaluation results with their uncertainty attached rather than as bare point estimates.

• Uncertainty-first reporting. Koji's analysis layer is built to accompany means, medians and top-box percentages with interval estimates, so a small-class result is presented as a range, not a deceptively exact number — the core discipline the bootstrap enables.
• Distribution-free by default. Because Koji reports the full response distribution, a reader can see the skew and ceiling that make normal-theory intervals unsafe, and resampling-based intervals are the natural companion to that distributional view.
• Honest small-n handling. Koji is designed to flag when response counts are too low to support reliable estimates rather than printing a confident-looking mean, and to lean on richer signals when numbers are thin — connecting to shrinkage/empirical-Bayes approaches for fair cross-course comparison.
• Qualitative depth where quantities run out. The deepest mitigation for a tiny, possibly biased sample is not a cleverer interval but better evidence. Koji's AI-moderated conversational interviews gather substantive open_ended reasoning from each respondent, and its automatic thematic analysis extracts patterns that a seven-person mean cannot carry — so a small class still yields interpretable insight rather than a fragile statistic.
• Anti-false-precision framing. Koji's reporting is designed to discourage ranking and year-over-year claims when intervals overlap, directly targeting the misclassification risk that motivates interval reporting in the first place. Koji is designed to mitigate false precision, not to manufacture certainty the data lack.

Koji's core research platform at koji.so applies the same uncertainty-aware, distribution-first reporting to product and customer research, where small-sample studies face identical over-interpretation risks.

Related Resources

• Small Mean Differences and Confidence Intervals in Course Evaluation
• Empirical Bayes Shrinkage for Instructor Course-Evaluation Scores
• How Many Responses Make a Course Evaluation Reliable?
• Funnel Plots for Fair Comparison of Instructor Scores
• Ceiling Effects and Skewed Course-Evaluation Distributions
• Ranking Instructors by Course-Evaluation Scores: Is It Fair?

A practical recipe for reporting

For a typical course-evaluation statistic the workflow is short. Take the observed responses; draw at least 2,000-10,000 resamples of the same size with replacement; recompute the statistic of interest - mean, median, or the top-box percentage - on each; and construct a BCa interval from the resulting distribution. Report the statistic with its interval and the response count together, for example "top-box 71% (95% CI 48-88%, n = 17)," so the reader sees the estimate, its uncertainty, and the base it rests on in a single line. Two habits keep the report honest. Present the interval before any comparison, so overlap is visible before a difference is asserted; and pair every interval with the response rate, because a tight interval on a low response rate is precise about a possibly unrepresentative sample. Modern statistical software computes BCa intervals in a single call, so the barrier to adoption is convention, not computation - which is precisely why false precision persists in evaluation reporting.

References

• Efron, B. (1979). Bootstrap methods: Another look at the jackknife. The Annals of Statistics, 7(1), 1–26. https://doi.org/10.1214/aos/1176344552
• Efron, B., & Tibshirani, R. J. (1993). An Introduction to the Bootstrap. Chapman & Hall/CRC.
• Efron, B., & DiCiccio, T. J. (1996). Bootstrap confidence intervals. Statistical Science, 11(3), 189–228. https://doi.org/10.1214/ss/1032280214
• Carpenter, J., & Bithell, J. (2000). Bootstrap confidence intervals: When, which, what? A practical guide for medical statisticians. Statistics in Medicine, 19(9), 1141–1164. https://doi.org/10.1002/(SICI)1097-0258(20000515)19:9<1141::AID-SIM479>3.0.CO;2-F