The Average Hides the Tails: Quantile Regression for Course-Evaluation Data

In brief

Ordinary least-squares regression and simple mean comparisons tell you what happened to the average student. Quantile regression (Koenker & Bassett, 1978) instead models chosen points of the whole outcome distribution — the 10th percentile, the median, the 90th — and can show that an intervention which barely moved the mean actually lifted the most satisfied students while leaving (or worsening) the least satisfied. For course evaluation, where the polarised minority often matters more for retention and equity than the average, modelling the tails rather than the centre frequently changes the conclusion.

What the research says

Quantile regression was introduced by Roger Koenker and Gilbert Bassett in a 1978 Econometrica paper, "Regression Quantiles". Where ordinary least squares (OLS) estimates the conditional mean of an outcome by minimising squared residuals, quantile regression estimates a conditional quantile (such as the median or the 90th percentile) by minimising an asymmetrically weighted sum of absolute residuals — the "check function", which penalises over- and under-prediction differently depending on the quantile being modelled. Estimating the median is the special symmetric case (least absolute deviations).

Koenker and Hallock's (2001) accessible Journal of Economic Perspectives introduction lays out why this matters: a single conditional-mean model assumes a predictor shifts the entire distribution by the same amount, whereas in reality an effect can differ across the distribution. Their canonical examples — the effect of parental characteristics on birth weight, of education on the spread of wages — show effects that are large at one tail and negligible at the other. Two properties make the method attractive as more than a curiosity:

• It characterises the whole conditional distribution, not just its centre, so heterogeneous effects (different at the top and bottom) become visible instead of being averaged into a single misleading number.
• It is robust to outliers in the outcome and makes no normality assumption. Because it works on absolute deviations and ranks rather than squared distances, a handful of extreme scores does not drag a quantile estimate the way it drags a mean.

Since 1978 quantile regression has become standard across economics, ecology, and the health sciences precisely because "the effect at the average" is often the least interesting part of the story.

Why it matters for course evaluation in practice

Course-evaluation distributions are rarely well-behaved bell curves. They are typically skewed, ceiling-bound (many 4s and 5s), and often bimodal — a satisfied majority and a distinct unhappy cluster. Reporting and comparing means on such distributions throws away exactly the information a quality process needs.

Consider a redesign intended to support struggling students. Evaluated by the mean, it might look like a wash: the average score barely moves. A quantile-regression view can reveal the real pattern — the 10th and 25th percentiles rose substantially (the intervention did help the students it targeted) while the 90th percentile dipped slightly (some high-flyers found the extra scaffolding tedious). The average effect was near zero because two real, opposite effects cancelled. That is a policy-relevant finding an OLS or mean comparison would have buried.

Three recurring course-evaluation questions are natural quantile-regression questions:

1. Equity and the unhappy tail. Did a change lift the students at the bottom of the satisfaction distribution — often the ones at risk of disengaging or dropping out — regardless of what it did to the mean?
2. Compression vs. shift. Did class size (or online delivery, or a new instructor) shift everyone equally, or did it mainly affect the spread — pulling the tails apart or squeezing them together? A mean cannot distinguish a shift from a change in dispersion.
3. Ceiling-bound comparisons. When most scores sit near the top, the median and lower quantiles carry the discriminating signal; modelling them directly is more honest than comparing means that are all pressed against the ceiling.

Quantile regression turns "on average, no difference" into a precise, distribution-aware statement about who was affected and how much.

Limitations and honest caveats

Quantile regression is a description of conditional quantiles, not a licence to over-interpret, and it carries real costs.

• The tails are noisy. Extreme quantiles (5th, 95th) are estimated from few effective observations and have wide confidence intervals — especially in small classes. A dramatic-looking effect at the 10th percentile of a 25-student module may be almost entirely sampling noise. Report interval estimates, ideally via the bootstrap, and resist reading a story into an unstable tail.
• It does not, by itself, establish causation. Like OLS, a quantile-regression coefficient is causal only under the same design assumptions (randomisation or a credible identification strategy). "The 10th percentile rose after the change" is still vulnerable to confounding, selection, and regression to the mean.
• Discreteness and heavy ceilings complicate it. Course ratings are coarse ordinal scales (often 1-5), and quantile regression was developed for continuous outcomes. With extreme clumping at the top, many quantiles land on the same value and estimates can be unstable; ordinal or cumulative-link models may be the better tool for a strictly ordinal single item.
• Interpretation is less intuitive. Stakeholders understand "the average went up 0.2". "The conditional 25th percentile increased by 0.4 while the 90th was unchanged" needs explanation, and a quantile that crosses another (the non-crossing problem) can confuse.
• Multiple quantiles invite multiple comparisons. Testing effects at many percentiles raises the false-positive rate; pre-specify the quantiles of interest rather than scanning for the one that is significant.

Used as a complement to the mean and reported with honest uncertainty, it is illuminating; used to cherry-pick the most flattering percentile, it misleads.

How Koji incorporates this

Koji is designed to keep the whole distribution — not a lone average — in front of the people making decisions, which is the mindset quantile regression formalises:

• Distribution-first reporting. Koji's analytics surface the full response distribution and its tails rather than collapsing a construct to a single mean, so a rise in the unhappy tail is visible even when the average is flat — the practical, everyday version of a quantile view.
• Explaining the tails, not just measuring them. Quantile regression tells you that the bottom of the distribution moved; it cannot tell you why. Koji's AI-moderated conversational interviews probe the low-satisfaction respondents specifically, converting a noisy 10th-percentile signal into an explanation a programme team can act on.
• Equity-aware segmentation. Because Koji captures structured attributes alongside open text, the "who is in the unhappy tail" question — the equity question quantile regression is built for — can be examined directly and paired with bias-aware reporting rather than inferred from a group mean.
• Honest uncertainty. For small cohorts, Koji's reporting emphasises that tail estimates are uncertain, discouraging the over-reading of a dramatic percentile that quantile-regression practice explicitly warns against.

We frame this as surfacing distributional signal for human judgement, not as automated causal inference — a quantile that moved is a lead to investigate, not a proven effect. Teams doing product and customer research can apply the same distribution-first thinking through Koji's core platform at koji.so, where the unhappy tail of users is often where the most valuable insight lives.

Frequently asked questions

What does quantile regression do that ordinary regression doesn't? Ordinary regression models the conditional mean and assumes a predictor shifts the whole distribution equally. Quantile regression models chosen points of the distribution (e.g., the 10th percentile, median, 90th), revealing effects that differ across the distribution and are hidden when you only look at the average.

When should I use it for course evaluations? When the tails matter more than the centre: checking whether a change helped the least satisfied students, distinguishing a shift in level from a change in spread, or comparing ceiling-bound distributions where means are all pressed against the top.

Is quantile regression robust to a few extreme scores? Yes — because it works on absolute deviations and ranks rather than squared distances, a handful of retaliatory 1s or 5s does not drag a quantile estimate the way it drags a mean. That robustness is a major reason to use it on skewed evaluation data.

Does it prove a teaching change caused the effect? No. Like any regression, a quantile coefficient is causal only under a sound design (randomisation or a credible identification strategy). It describes the conditional distribution; it does not by itself rule out confounding or regression to the mean.

Can I trust an effect at the 10th percentile of a small class? Be cautious. Extreme quantiles are estimated from few effective observations and have wide confidence intervals, so tail effects in small modules are often mostly noise. Report bootstrap intervals and pre-specify which quantiles you care about.

Is quantile regression appropriate for a 1-5 Likert item? With care. It was built for continuous outcomes; heavy clumping on a coarse ordinal scale can make many quantiles land on the same value and destabilise estimates. For a strictly ordinal single item, an ordinal/cumulative-link model is often the better choice.

Related resources

• Ordinal Regression for Course-Evaluation Data: Cumulative-Link Models
• When a Few Retaliatory 1s Sink the Average: Robust Estimators
• When Almost Everyone Scores 4.5: Ceiling Effects, Skew, and Statistics
• Top-Box vs Mean: Reporting Without Throwing Away Information
• Students Are Nested in Courses: Why Multilevel Models Beat Raw Averages
• Should You Average a 1-5 Course-Evaluation Scale at All?

References

• Koenker, R., & Bassett, G. (1978). Regression quantiles. Econometrica, 46(1), 33-50. https://doi.org/10.2307/1913643
• Koenker, R., & Hallock, K. F. (2001). Quantile regression. Journal of Economic Perspectives, 15(4), 143-156. https://doi.org/10.1257/jep.15.4.143
• Koenker, R. (2005). Quantile Regression. Cambridge: Cambridge University Press. https://doi.org/10.1017/CBO9780511754098
• Cade, B. S., & Noon, B. R. (2003). A gentle introduction to quantile regression for ecologists. Frontiers in Ecology and the Environment, 1(8), 412-420. https://doi.org/10.1890/1540-9295(2003)001[0412:AGITQR]2.0.CO;2