Changepoint Detection for Course-Evaluation Time Series

In brief

A course's rating series often drifts, and the useful question is not merely "did it change" but "when did it change" — when you do not know the date in advance. Changepoint detection finds the point or points in a time-ordered sequence where the statistical behaviour shifts — the mean, the variance, or the whole distribution — without you specifying the location beforehand. It is the retrospective, unknown-location counterpart to prospective control-chart monitoring and to interrupted-time-series analysis of a known intervention, and it turns a wobbly line of term-by-term scores into a small set of dated, testable structural breaks.

What the research says

The idea grew out of industrial quality control. Page (1954), "Continuous inspection schemes," introduced the CUSUM (cumulative sum) scheme for detecting a shift in a process mean as observations accumulate. Modern offline changepoint detection frames the task as choosing the number and location of changes that minimise a total cost — for example the summed within-segment variance — plus a penalty for each added change, so the method does not simply carve the series at every wiggle. Exhaustive search over all possible segmentations is exponential, which long made multiple-changepoint detection impractical. Killick, Fearnhead and Eckley (2012) solved this with PELT (Pruned Exact Linear Time), an algorithm that finds the exact optimal segmentation in roughly linear time by pruning candidate change locations that can never be optimal; binary segmentation remains a fast approximate alternative.

Truong, Oudre and Vayatis (2020) give a selective review that organises every offline method by three choices: the cost function (what kind of change you are looking for — a shift in mean, in variance, or in a fuller distribution), the search method (exact like PELT, or approximate like binary segmentation), and the constraint (a fixed number of changes versus a penalty that lets the data decide). In econometrics the identical problem appears as "structural breaks"; Bai and Perron (1998) provide the canonical framework for estimating and testing multiple structural changes in a regression relationship, including how to test whether a break exists at all. Together these give a principled, well-tooled way to answer "how many shifts, and where."

Why it matters for course evaluation in practice

Longitudinal course and instructor series — a mean rating each term for years — constantly prompt the question "the score dropped, so when did that start and what changed?" Eyeballing a jagged line invites seeing turning points that are only noise.

It dates the break objectively. Changepoint detection places the shift at a defensible location with uncertainty around it, rather than leaving a committee to argue over which term "felt" like the change.
It separates a level shift from wobble. A genuine drop after a curriculum redesign whose exact timing nobody recorded looks different from term-to-term sampling noise, and complements regression to the mean, which explains why a low term usually rebounds on its own.
It catches variance changes, not just means. A programme whose ratings became more polarised while the average held is a real change a mean-only view would miss.
It is distinct from its siblings. Statistical process control charts monitor prospectively against control limits for out-of-control signals; interrupted time series tests a change at a known intervention date; changepoint detection discovers an unknown date retrospectively.

Limitations and honest caveats

Changepoint detection needs a reasonable series length — a handful of terms cannot support reliable detection, and breaks found near the start or end of a series are unstable. The penalty choice (BIC, a modified BIC, or a manual threshold) largely determines how many changepoints you get: set it too low and the method over-segments noise into phantom shifts; set it too high and it misses real ones, so a sensitivity analysis across penalties is mandatory, not optional. Autocorrelation and seasonality — the cyclical cohort structure of an academic calendar — can masquerade as changepoints, so the series should be modelled or de-seasonalised first. Detection is descriptive: a dated break is a hypothesis about when the behaviour changed, never evidence of why, so attribution still requires design and context, for instance a difference-in-differences comparison against an unaffected course. Finally, small class sizes make each term's mean noisy, widening the uncertainty around any detected break and sometimes making detection impossible with honest error bars.

How Koji incorporates this

Koji's longitudinal reporting treats a course's term-by-term ratings as a time series rather than a set of disconnected snapshots, so it can flag a dated level or variance shift as a candidate structural break instead of leaving reviewers to eyeball a wavy line. Koji is designed to expose the penalty and sensitivity behind any detected changepoint — showing how the number of breaks changes as the threshold moves — so a committee does not over-read noise as a turning point, and it distinguishes a shift in the mean from a shift in the variance, surfacing the case where ratings simply became more polarised. Because Koji collects continuously through AI-moderated conversational interviews, a detected break can be linked to the open-text themes clustered around that term, giving reviewers the why to test against the when rather than guessing. Koji deliberately keeps changepoint detection separate from its prospective control-chart monitoring and from interrupted-time-series analysis of a known change date, so the right tool answers each distinct question. These mechanisms are built to date shifts honestly, not to certify their cause, which still requires context and design on any platform. Teams tracking longitudinal product or customer metrics can apply the same changepoint logic in Koji's core research platform at koji.so, where a satisfaction series raises the identical "when did it move" question.

A practical workflow

In practice, dating a shift in a course's ratings follows a short sequence. First, assemble a long-enough series — ideally a dozen or more consecutive offerings, each with its mean and its response count, because a break inferred from four terms is little more than a guess. Second, remove predictable structure: strip any regular autumn-versus-spring cohort difference and check for autocorrelation, so the method does not read the calendar as a change. Third, run a penalised search such as PELT for a shift in the mean, and separately for a shift in the variance, since a programme can hold its average while its ratings pull apart. Fourth — the step teams most often skip — sweep the penalty across a range and watch how the number of breaks responds: a changepoint that appears only at the most permissive penalty and vanishes as the threshold tightens is noise wearing a costume. Fifth, put an uncertainty window around each surviving break rather than a single date, because the exact term is rarely pinned down precisely. Only then bring in context — the redesign, the staffing change, the room move — and test the why against the dated when. Following this order keeps a genuine structural break, such as the term a modular redesign took effect, clearly distinct from the ordinary wobble that a wavy line always shows.

Frequently asked questions

What is changepoint detection?

It is a family of methods that find where, in a time-ordered series, the statistical behaviour shifts — a change in the mean, the variance, or the distribution — without you specifying the location in advance. It answers "how many shifts happened, and when."

How is it different from a control chart?

A control chart (statistical process control) monitors data prospectively as it arrives and signals when a point breaches control limits. Changepoint detection works retrospectively on a completed series to locate one or more shifts whose timing was unknown, and can date breaks that a chart only flags as they occur.

How is it different from interrupted time series?

Interrupted time series tests whether an outcome changed at a known intervention date that you supply. Changepoint detection discovers the date, which is what you need when a shift happened but nobody recorded exactly when the cause took effect.

How do I avoid finding fake changepoints?

Use a penalty that charges for each added change, run a sensitivity analysis across penalty values, and first remove seasonality and autocorrelation, which can mimic breaks. Also require a reasonable series length and treat breaks near the ends of the series with caution.

Can it detect a change in variability, not just the average?

Yes. By choosing a cost function tuned to variance, changepoint detection can flag a term where ratings became more polarised even though the mean barely moved, a pattern a mean-only comparison would miss entirely.

Does a detected changepoint tell me why the rating shifted?

No. It dates the shift, which is a hypothesis about timing. Establishing the cause needs context and a design such as difference-in-differences against an unaffected comparison course, plus the qualitative themes around that period.

References

Page, E. S. (1954). Continuous Inspection Schemes. Biometrika, 41(1/2), 100-115. https://doi.org/10.1093/biomet/41.1-2.100
Killick, R., Fearnhead, P., & Eckley, I. A. (2012). Optimal Detection of Changepoints With a Linear Computational Cost. Journal of the American Statistical Association, 107(500), 1590-1598. https://doi.org/10.1080/01621459.2012.737745
Truong, C., Oudre, L., & Vayatis, N. (2020). Selective review of offline change point detection methods. Signal Processing, 167, 107299. https://doi.org/10.1016/j.sigpro.2019.107299
Bai, J., & Perron, P. (1998). Estimating and Testing Linear Models with Multiple Structural Changes. Econometrica, 66(1), 47-78. https://doi.org/10.2307/2998540

When Did the Rating Actually Shift? Changepoint Detection for Course-Evaluation Time Series