When a Few Retaliatory 1s Sink the Average: Robust Estimators for Course Evaluation

In brief

In a small class, a handful of extreme ratings — two retaliatory 1s, one ecstatic 5 — can swing the reported mean by half a point or more, and high-stakes decisions get made on that swing. The arithmetic mean has a breakdown point of zero: a single arbitrary value can move it arbitrarily far. Robust estimators of central tendency — the trimmed mean, Winsorized mean, and M-estimators — are designed to resist exactly this, summarising the bulk of the responses without letting a few extreme or careless ones dominate, and without discarding whole respondents. For the small, skewed, bounded distributions typical of course evaluation, robust summaries are often a fairer statement of "how this class went" than the raw mean, provided you report which quantity you are estimating.

What the research says

The problem is old and well characterised. Huber (1964), in the Annals of Mathematical Statistics, launched modern robust statistics with "Robust Estimation of a Location Parameter", introducing M-estimators — estimators that down-weight observations far from the centre instead of letting them count in full. The key concept is the breakdown point: the fraction of arbitrarily bad data an estimator can tolerate before it, too, becomes arbitrarily bad. The mean breaks down at 0%; the median at 50%; a 20% trimmed mean at 20%.

Wilcox and Keselman (2003), in Psychological Methods, reviewed decades of this work for behavioural researchers and made the practical case bluntly: conventional methods built on means, t tests, and ANOVA lose both accuracy and statistical power when data are non-normal or contaminated by outliers — conditions that describe most real datasets. They advocate robust measures of location, particularly trimmed means (discard a fixed percentage from each tail and average the rest) and M-estimators, paired with robust inference such as Yuen's test, which compares trimmed means using a Winsorized variance for the standard error. Erceg-Hurn and Mirosevich (2008), in American Psychologist, reinforced the message: the normality and equal-variance assumptions classical tests require are rarely met, and modern robust methods are an easy way to protect the accuracy and power of applied research.

On the practical question of spotting the extreme values, Leys, Ley, Klein, Bernard, and Licata (2013), in the Journal of Experimental Social Psychology, argue that the usual "mean ± so many standard deviations" rule is self-defeating because outliers inflate the very standard deviation used to detect them. They recommend the median absolute deviation (MAD) — a robust measure of spread — with a flag at roughly the median ± 2.5 MAD. Together these papers describe a coherent toolkit: robust location (trimmed mean, M-estimator), robust spread (MAD), and robust inference (Yuen).

Why it matters for course evaluation in practice

Course-evaluation distributions violate classical assumptions in almost every way that matters: cohorts are small, the scale is bounded and coarse (1–5), distributions are skewed and ceiling-heavy, and a minority of responses are careless or retaliatory. In that setting the arithmetic mean is fragile. Consider a class of twelve with ten ratings of 4 and two of 1: the mean is 3.5, a full point below where nine-tenths of the class sits, purely because of two extreme values that the mean weights as heavily as everyone else. A 20% trimmed mean would set aside the two most extreme responses at each end and report roughly 4 — a truer summary of the typical experience.

Robust summaries change three practical things. Reporting: publishing a trimmed mean alongside the raw mean (and top-box) tells a fairer story of the central experience, and a divergence between them is itself a signal — it flags a polarised class or a few influential responses worth reading rather than averaging away. Screening: the MAD rule gives a defensible, outlier-resistant way to flag unusual responses for inspection before any decision. Comparison: when comparing two instructors or sections on small samples, Yuen's trimmed-means test is less likely than the ordinary t test to be thrown by a single outlier or to lose power under skew. For personnel-adjacent uses especially, resisting the influence of a couple of extreme ratings is not a statistical nicety — it is basic fairness to the instructor.

Limitations and honest caveats

Robustness is not free, and a careful reader should press on four points. First, trimming changes the estimand. A 20% trimmed mean estimates a different population quantity than the ordinary mean; on a symmetric distribution they coincide, but on a skewed one they do not, and you must state which you are reporting rather than swapping them silently. Second, on a bounded 1–5 scale an "outlier" may be a legitimate strong opinion, not an error. A genuine, furious 1 from a student who was badly let down is real information; robust methods that quietly down-weight it can mask real polarisation or a real minority harm. The right move is to inspect flagged responses — often via their open text — not to trim reflexively. Third, the trimming fraction is a choice. Twenty percent is a well-studied convention, but it is a decision that should be pre-specified, not tuned until the result looks good. Fourth, robust ≠ correct model. For ordinal Likert data, many methodologists argue the deeper fix is an ordinal (cumulative-link) model rather than any mean-like summary at all; robust location estimators are a pragmatic improvement on the mean, not a resolution of the ordinal-versus-interval debate. Two smaller cautions: with very small cohorts (under ~10) there is little data left after trimming, so robust estimators have limited traction; and the bare median, while maximally robust, discards a lot of information and produces many ties on a coarse scale, which is why trimmed means and M-estimators are usually preferred over it for evaluation reporting. Finally, robust estimation is not a substitute for genuine data-quality screening — it limits the influence of bad responses but does not diagnose why they are there.

How Koji incorporates this

Koji treats robust summaries as a reporting option to sit beside the mean, coupled with the context needed to decide whether down-weighting an extreme response is fair:

• Robust summaries alongside the mean. Koji's reporting can present a trimmed mean and MAD-based spread next to the arithmetic mean and top-box score, and highlight when they diverge — a divergence that tells a QA reader "read this class more closely" rather than trusting a single number.
• Flag, then read — not silent trimming. Because a course-evaluation "outlier" can be a legitimate strong view, Koji pairs MAD-style flagging with the student's own open text and its automatic thematic analysis, so a reviewer can see why a 1 was given before deciding whether it reflects a data-quality problem or a real, serious concern.
• Distinguishing careless from committed. Koji's insufficient-effort/careless-response screening helps separate genuinely uninformative responses (straightlining, ten-second completions) from strongly-felt extreme ratings, so robust down-weighting targets noise rather than dissent.
• Transparent and pre-specified. Koji is built to let institutions fix the trimming fraction and reporting rules in advance, so robust methods are applied as consistent policy, not chosen after seeing the scores.

The same reporting engine underlies Koji's core research platform at koji.so, where product and customer teams face the identical problem of a few extreme responses distorting a small-sample average.

Frequently asked questions

What is a trimmed mean, concretely? You discard a fixed percentage of responses from each tail — commonly 20% — and average the rest. In a class of ten with a 20% trim you drop the two highest and two lowest ratings and average the middle six, so a couple of extreme values cannot dominate the summary.

How is Winsorizing different from trimming? Trimming removes the extreme responses; Winsorizing replaces them with the nearest retained value, keeping the sample size the same. The Winsorized variance is what robust tests such as Yuen's use to compute a fair standard error for the trimmed mean.

Is a robust estimator just the median? The median is one robust estimator, with the highest breakdown point, but it discards a lot of information and produces many ties on a coarse 1–5 scale. Trimmed means and M-estimators keep more of the data while still resisting outliers, which is usually preferable for evaluation reporting.

Should I always trim course-evaluation data? No. Trimming changes what you are estimating, and on a bounded scale an extreme rating can be a legitimate strong opinion rather than an error. Report robust and ordinary summaries together, inspect flagged responses, and pre-specify your rule rather than trimming reflexively.

How is this different from using the bootstrap? The bootstrap estimates the sampling distribution and confidence interval of a statistic; robust estimators change which statistic you compute so it resists outliers. They are complementary — a common best practice is to bootstrap a trimmed mean.

Related resources

• The Bootstrap for Small-Class Course-Evaluation Reporting
• Ceiling Effects, Skew, and Course-Evaluation Statistics
• Ordinal Regression for Course-Evaluation Likert Data
• Empirical-Bayes Shrinkage for Course-Evaluation Scores
• Top-Box vs Mean: Reporting Without Throwing Away Information
• Detecting Careless and Insufficient-Effort Responding

References

• Huber, P. J. (1964). Robust Estimation of a Location Parameter. The Annals of Mathematical Statistics, 35(1), 73–101. https://doi.org/10.1214/aoms/1177703732
• Wilcox, R. R., & Keselman, H. J. (2003). Modern Robust Data Analysis Methods: Measures of Central Tendency. Psychological Methods, 8(3), 254–274. https://doi.org/10.1037/1082-989X.8.3.254
• Erceg-Hurn, D. M., & Mirosevich, V. M. (2008). Modern Robust Statistical Methods: An Easy Way to Maximize the Accuracy and Power of Your Research. American Psychologist, 63(7), 591–601. https://doi.org/10.1037/0003-066X.63.7.591
• Leys, C., Ley, C., Klein, O., Bernard, P., & Licata, L. (2013). Detecting Outliers: Do Not Use Standard Deviation Around the Mean, Use Absolute Deviation Around the Median. Journal of Experimental Social Psychology, 49(4), 764–766. https://doi.org/10.1016/j.jesp.2013.03.013