When Can You Stop Collecting? Optional Stopping and Sequential Analysis for Rolling Course Evaluation

In brief

If you watch course-evaluation results accumulate in real time and decide to stop collecting — or to declare an instructor difference "significant" — the moment the p-value dips below 0.05, you have inflated your false-positive rate, often dramatically. This is the optional-stopping (or "peeking") problem, and it is a genuine risk for any always-on, dashboard-driven evaluation platform. The fix is not to ban looking; it is to plan the looks. Sequential-analysis methods — group-sequential boundaries, alpha-spending, or sequential Bayes factors — let you monitor accumulating data and stop early while keeping error rates controlled. The catch is that these methods require pre-specification and discipline; used informally they give false reassurance.

What the research says

The statistical hazard was quantified decades ago. Armitage, McPherson and Rowe (1969), in the Journal of the Royal Statistical Society, showed formally that repeating a fixed-level significance test as data accumulate drives the probability of a false positive well above the nominal level: test after every new batch of observations and, under a true null, you are almost guaranteed to eventually cross the 0.05 line. The error does not stay at 5%; it grows with the number of looks.

Simmons, Nelson and Simonsohn (2011), in Psychological Science, made the practical consequences vivid. Among the "researcher degrees of freedom" they simulated, collecting data until a result is significant and then stopping was one of the most corrosive: in their simulations, flexible stopping alone pushed the false-positive rate substantially above 5%, and combined with other minor flexibilities it exceeded 60%. Their prescription — decide the sample size (or the stopping rule) in advance and report it — is now standard.

The constructive answer is sequential analysis, whose lineage runs from Wald (1945) and his sequential probability ratio test through the group-sequential designs of clinical trials. Lakens (2014), in the European Journal of Social Psychology, brought these tools to the behavioural sciences: with pre-specified interim analyses and an alpha-spending function, you can look at the data several times, stop as soon as the evidence is convincing, and still hold the overall Type I error at 5%. The trade-off is a modestly higher per-look threshold — the "cost" of each peek is paid explicitly rather than ignored. Classic boundary schemes such as Pocock's constant threshold and the O'Brien-Fleming spending function (which is very strict early and relaxes near the planned end) are the standard ways to allocate that cost.

Why it matters for course evaluation in practice

Traditional end-of-term evaluation collected everything at once and analysed it once, so optional stopping rarely arose. Modern, continuous collection changes that. When responses stream into a live dashboard, the temptation to act the moment a comparison "goes significant" — to flag an instructor, to stop a pilot early, to announce that a redesign worked — is constant, and acting on it manufactures false alarms.

This connects directly to how many responses you actually need. Deciding the target in advance, as covered in how many responses make an evaluation reliable and a-priori power analysis, is the first line of defence: a pre-registered sample size removes the incentive to peek-and-stop. Where genuine ongoing monitoring is the goal — tracking a metric over time rather than testing a single hypothesis — statistical process control charts are designed for exactly that repeated looking, with control limits set so that routine variation does not trip an alarm. And the same multiplicity logic that governs peeking over time governs comparing many instructors at once, the subject of multiple comparisons and the false discovery rate. Both are failures to account for how many chances you gave yourself to find something.

Limitations and honest caveats

The sophisticated reader should not treat sequential methods as a licence to look freely.

Pre-specification is the whole point. Group-sequential and alpha-spending methods control error only when the number and timing of looks (or the spending function) are fixed in advance. Inventing the stopping rule after you start looking reintroduces exactly the bias you were trying to remove.

Early stopping biases effect sizes upward. Studies that stop early "for success" tend to overestimate the effect, because you are most likely to cross the boundary on a favourable random swing. A difference that triggered an early stop should be reported with that inflation in mind and, ideally, shrunk.

Course evaluation is rarely a clean hypothesis test. Much evaluation use is descriptive monitoring, not a two-arm confirmatory test. Forcing it into a group-sequential trial framework can be a category error; sometimes a control chart or a Bayesian posterior that is simply reported (not thresholded) is the honest tool.

Bayesian sequential methods are not automatically immune. Sequential Bayes factors let you stop when evidence is strong, and a Bayes factor is not a p-value — but naively stopping at a fixed Bayes-factor threshold still changes the sampling properties of the procedure, and should be planned and simulated, not improvised.

Small cohorts limit everything. Interim analyses need enough data at each look to be meaningful. In a seminar of twenty, there is no useful "sequence" to analyse.

How Koji incorporates this

Koji's model of collection — running across the term rather than only at the end — is precisely the setting where optional stopping can bite, so the platform is designed to make responsible practice the default rather than an afterthought. Live response tracking is treated as monitoring, distinct from hypothesis testing: watching completion and coverage accumulate is encouraged, while any inferential comparison is framed against a pre-specified target sample rather than invited to be stopped the instant a threshold is crossed.

Concretely, several Koji mechanisms map onto the research. The platform supports setting a target sample or response-rate goal up front, which removes the peek-and-stop incentive that Simmons and colleagues identified. Its reporting is designed to distinguish stable, adequately-powered comparisons from noise — small differences on thin samples are flagged as such rather than presented as findings, echoing the caution behind the 4.2-versus-4.4 problem. And because Koji triangulates numeric scores with AI-moderated conversational interviews and automatic thematic analysis of open text, a decision does not hinge on a single accumulating p-value; qualitative corroboration provides an independent check on whether an early signal is real.

We frame these as safeguards designed to reduce peeking-driven false positives, not as a mathematical guarantee: the discipline of pre-specifying a stopping rule still rests with the evaluation office. Koji's core research platform at koji.so applies the same AI-moderated interview engine to product and customer research, where continuous data streams make the optional-stopping trap just as live and the same pre-specification discipline applies.

Related resources

• How Many Responses Do You Need for a Reliable Course Evaluation?
• How Big a Difference Can You Actually Detect? A-Priori Power Analysis
• Statistical Process Control for Course Evaluations: Control Charts
• Compare 60 Instructors, Expect 3 False Alarms: Multiple Comparisons and the FDR
• Is There Really No Difference? Bayes Factors for Course-Evaluation Comparisons

Frequently asked questions

What exactly is the optional-stopping problem?

Optional stopping is deciding when to stop collecting data based on the results so far — for example, halting the moment a comparison becomes statistically significant. Because each look is another chance to cross the significance line by luck, repeated peeking inflates the false-positive rate well above the nominal 5%, and under a true null a persistent peeker will almost always eventually find "significance."

Does this mean I should never look at results while they come in?

No. The problem is not looking; it is stopping or deciding based on unplanned looks. You can monitor coverage and completion freely, and for inferential decisions you can use sequential-analysis methods that pre-specify the number and timing of looks so that error rates stay controlled while still allowing early stopping.

How do sequential analyses keep the error rate under control?

They allocate the total allowable Type I error across the planned interim analyses using an alpha-spending function or fixed boundaries (such as Pocock or O'Brien-Fleming). Each look uses a stricter threshold than 0.05, so the cost of peeking is paid explicitly. If the evidence crosses the boundary, you may stop early; otherwise you continue to the planned end.

Are Bayesian methods immune to peeking?

Not automatically. A Bayes factor is not a p-value and sequential Bayes factors are a legitimate way to stop when evidence is strong, but naively stopping at a fixed Bayes-factor threshold still alters the procedure's sampling behaviour. Bayesian stopping rules should be planned and simulated rather than improvised.

Why is a difference that triggered an early stop often overestimated?

Because you are most likely to cross an early-stopping boundary during a favourable random fluctuation. Selecting the moment the effect looks largest biases the estimate upward, so effect sizes from early-stopped analyses should be interpreted cautiously and, where possible, shrunk toward the pooled estimate.

Is optional stopping even relevant if my class is small?

Much less so, because interim analyses need enough data at each look to mean anything. In a seminar of twenty there is no meaningful sequence to analyse. Sequential methods matter most for programme- or cohort-level monitoring where responses accumulate in large numbers over time.

References

• Armitage, P., McPherson, C. K., & Rowe, B. C. (1969). Repeated significance tests on accumulating data. Journal of the Royal Statistical Society: Series A (General), 132(2), 235-244. https://doi.org/10.2307/2343787
• Wald, A. (1945). Sequential tests of statistical hypotheses. Annals of Mathematical Statistics, 16(2), 117-186. https://doi.org/10.1214/aoms/1177731118
• Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359-1366. https://doi.org/10.1177/0956797611417632
• Lakens, D. (2014). Performing high-powered studies efficiently with sequential analyses. European Journal of Social Psychology, 44(7), 701-710. https://doi.org/10.1002/ejsp.2023