Your Ratings Have a Term Rhythm: STL Seasonal-Trend Decomposition for Rolling Course Evaluation

In brief

When course feedback is collected continuously rather than in one end-of-term burst, the resulting rating series carries a repeating calendar rhythm — an autumn-intake effect, an exam-period dip, a January re-start — sitting on top of a slow underlying trend and a layer of noise. Comparing this quarter to last quarter, or this term to last, confounds a genuine change with the seasonal swing. STL (Seasonal-Trend decomposition using Loess) splits a rating series into three additive components — trend, season, and remainder — so you can read the direction of travel without being fooled by the time of year, and spot an anomaly that the calendar cannot explain.

What the research says

STL was introduced by Cleveland, Cleveland, McRae and Terpenning (1990) in the Journal of Official Statistics. It decomposes a time series Y(t) into three additive parts, Y(t) = T(t) + S(t) + R(t): a smooth trend T, a seasonal component S that captures the repeating within-year (or within-term) pattern, and a remainder R that is what is left once trend and season are removed. The method works by repeatedly applying loess — locally weighted regression, the robust scatterplot smoother Cleveland (1979) had introduced a decade earlier — in an inner loop that estimates the seasonal and trend components and an outer loop that down-weights outliers. Three properties make STL well suited to evaluation data. It allows the seasonal pattern to change gradually over time rather than forcing it to be identical every year; it is robust to occasional aberrant points (a cluster of retaliatory ones, a scandal week) so that those do not distort the trend; and it is computationally cheap, so long or high-frequency series decompose quickly.

STL sits within a much older tradition of seasonal adjustment used by national statistical agencies. Hyndman and Athanasopoulos, in the widely used open textbook Forecasting: Principles and Practice, present STL alongside classical additive and multiplicative decomposition and the X-11/X-13 family, and give the standard practical guidance: choose additive decomposition when the size of the seasonal swing is roughly constant, and a multiplicative form (or a log transform) when the swing grows with the level of the series. Their treatment also stresses the honest precondition — you need several complete cycles of data before a seasonal component can be estimated at all, and the trend at the very start and end of a series is the least certain part of the decomposition.

Why it matters for course evaluation in practice

Traditional evaluation is a once-a-term census, so there is no series to decompose. But always-on and mid-cycle collection — pulse questions, rolling conversational check-ins, continuous feedback widgets — produce exactly the kind of ordered, repeating-calendar data STL was built for. Three practical mistakes follow from ignoring seasonality. The first is spurious alarm: a dashboard shows satisfaction falling from November to December and a manager concludes teaching has deteriorated, when in fact every December dips as coursework deadlines and exam stress peak. The second is masked improvement: a real gain from a teaching redesign is hidden because it happens to land in a naturally low part of the calendar, so the raw number looks flat. The third is misattributed comparison: benchmarking an autumn-entry cohort against a spring-entry one treats a seasonal difference as a quality difference.

STL addresses all three by giving you the trend component — the series with the calendar rhythm and the noise stripped out — as the thing to watch and report. Month-to-month or term-to-term change is then read off the trend, not the raw line. The seasonal component is itself informative: it quantifies how much of the observed variation is simply "the time of year," which is useful when explaining to a committee why a December score should not be compared with a March one. And the remainder is a clean anomaly detector: once trend and season are removed, an unusually large residual flags a week or cohort that neither the underlying trajectory nor the calendar can account for, which is precisely where a quality office should look.

Limitations and honest caveats

STL is a descriptive decomposition, not a causal model, and it inherits the limits of the data feeding it. It needs several full cycles to estimate a stable seasonal pattern; with barely two years of monthly data, or two terms of weekly data, the seasonal and trend estimates are fragile and can absorb each other. The decomposition is also sensitive to tuning — the seasonal and trend smoothing windows are analyst choices, and different settings can shift how much variation is assigned to trend versus season, so the components are not uniquely determined. Its additive form assumes the seasonal swing is roughly constant in size; when the swing scales with the level, an untransformed additive STL will mis-split the series. The endpoints are the least reliable region, which is unfortunate because the most recent point is usually the one managers care about most; the current-quarter trend estimate should always be treated as provisional and revised as more data arrive. Finally, STL assumes a genuinely regular calendar rhythm; irregular academic calendars, changing collection instruments, or shifting response populations can create pseudo-seasonality that STL will faithfully — and misleadingly — model. It is a lens for seeing structure, not a substitute for knowing your own collection process.

How Koji incorporates this

Because Koji supports continuous and mid-cycle collection rather than only a single end-of-term survey, the data it gathers naturally form the ordered series that decomposition needs. Koji's reporting is built to separate the trend a programme director should act on from the seasonal rhythm that is simply the shape of the academic year, so a December dip is shown in context rather than raised as an alarm. The robust core of the method matters here: because Koji already screens for careless and retaliatory responses, and because loess-based decomposition down-weights the outliers that survive, a handful of extreme weeks does not drag the underlying trend. The remainder view doubles as an early-warning signal — a residual that the calendar and the trajectory cannot explain is exactly the kind of case Koji surfaces for a closer, qualitative look through its AI-moderated interviews and automatic thematic analysis, so the numeric anomaly is paired with the why. Koji frames all of this as provisional at the current edge of the series, in line with the endpoint caveat above, so recent movement is presented as a signal to watch rather than a settled verdict. The same time-aware reporting powers Koji's core research platform at koji.so, where product and customer-research teams running continuous feedback face identical calendar effects — seasonal buying cycles, release rhythms — that must be separated from real change.

Frequently asked questions

How is STL different from a control chart or statistical process control?

Statistical process control monitors a process against fixed control limits and assumes, in its basic form, a stable mean with no systematic calendar pattern. STL explicitly models and removes the calendar pattern first. In practice they are complementary: decompose with STL, then apply control-chart logic to the remainder. Our note on statistical process control covers the monitoring side.

How is this different from changepoint detection or interrupted time series?

Changepoint detection looks for an abrupt, unknown-location break in a series, and interrupted time series estimates the effect of a change at a known date. STL does neither — it separates the recurring seasonal rhythm from the trend so that those other methods are not fooled by seasonality masquerading as a break. See changepoint detection and interrupted time series.

How much data do I need before STL is trustworthy?

As a rule of thumb, at least two and preferably three or more complete seasonal cycles. With a weekly series and a term-length season you need several terms; with a monthly series and an annual rhythm you need a few years. Below that, treat the seasonal component as tentative.

Should I use additive or multiplicative decomposition?

Use additive when the size of the seasonal swing stays roughly constant regardless of the overall level. Use multiplicative (or take logs and use additive) when the swing grows as ratings rise or fall. For bounded 1–5 rating means the swing is usually modest and additive is a reasonable default.

Can STL handle the messy, irregular academic calendar?

Partly. STL assumes a regular period. Irregular term lengths, moved reading weeks, or a changed survey instrument can create structure STL will model as "seasonal" even though it is really an artefact. Align your series to a consistent cadence and note instrument changes so you can tell genuine seasonality from process quirks.

Does the most recent point get revised?

Yes, and this is expected. The trend estimate at the end of a series is the least certain and will move as new data arrive. Report the current edge as provisional and avoid basing an irreversible decision on a single freshly decomposed endpoint.

References

• Cleveland, R. B., Cleveland, W. S., McRae, J. E., & Terpenning, I. (1990). STL: A Seasonal-Trend Decomposition Procedure Based on Loess. Journal of Official Statistics, 6(1), 3–33. (Open access; Statistics Sweden / Journal of Official Statistics archive.)
• Cleveland, W. S. (1979). Robust Locally Weighted Regression and Smoothing Scatterplots. Journal of the American Statistical Association, 74(368), 829–836. https://doi.org/10.1080/01621459.1979.10481038
• Hyndman, R. J., & Athanasopoulos, G. (2021). Forecasting: Principles and Practice (3rd ed.). OTexts. https://otexts.com/fpp3/decomposition.html

Related resources

• When Did the Rating Actually Shift? Changepoint Detection for Course-Evaluation Time Series
• Statistical Process Control for Course Evaluations: Using Control Charts to Separate Signal from Noise
• Did the Teaching Change Actually Work? Interrupted Time Series for Course-Evaluation Trends
• When Can You Stop Collecting? Optional Stopping and Sequential Analysis for Rolling Course Evaluation
• Should You Run Course Evaluations Before or After the Final Exam? The Timing Evidence
• When a Few Retaliatory 1s Sink the Average: Robust Estimators for Course Evaluation