Multi-Armed Bandits and Adaptive Experiments in Course Evaluation

In brief

A multi-armed bandit is a rule for allocating a limited number of trials across several options — the "arms" — when you do not yet know which is best, balancing exploration (learning which arm wins) against exploitation (using the apparent winner). In course evaluation the arms are competing question wordings, follow-up probe strategies, reminder messages or survey formats, and an adaptive experiment shifts more respondents toward the better-performing arm as evidence accrues instead of locking a fixed 50/50 split. The catch, established directly in the education literature, is that the same adaptivity that benefits students distorts the statistics you get out: bandits buy engagement and participant welfare at a measurable cost to clean inference.

What the research says

The problem was named by Robbins (1952), "Some aspects of the sequential design of experiments," which framed the tension between gathering information and acting on it. The elegant Bayesian solution — allocate each option in proportion to the posterior probability that it is the best — traces to Thompson (1933) and is now called Thompson sampling. Scott (2010) gives the modern Bayesian treatment, "randomized probability matching," and shows it performs strongly against alternatives for online business experiments; Russo and colleagues (2018) provide the standard tutorial on when and why Thompson sampling works.

The decisive evidence for our setting is Rafferty, Ying and Williams (2019), who studied bandits specifically for adaptive educational experiments in the Journal of Educational Data Mining. Multi-armed-bandit algorithms send more students to the better-performing condition, which benefits learners, but the authors showed that the data such algorithms produce inflates false-positive rates and biases the estimated effect, so a naive significance test on adaptively collected data is untrustworthy. They also showed that temporal trends in student behaviour — differences between early-term and late-term respondents — can seriously mislead a bandit, because the algorithm may lock onto an arm that only looked best during an unrepresentative early window. The core finding is a genuine trade-off: reward for participants versus statistical reliability.

Why it matters for course evaluation in practice

Evaluation teams already run informal A/B tests without naming them — two subject lines for the reminder email, two wordings of a sensitive item, a conversational format versus a static form. A bandit formalises the instinct to stop wasting scarce responses on the worse option, which is valuable precisely because course-evaluation responses are limited and non-renewable.

Adaptive probing is on-brand for modern tools. Which follow-up prompt elicits the richest open text? A bandit can concentrate collection on the better prompt while the study is still running, raising the quality of the feedback gathered.
The goal decides the design. If you need a clean confirmatory estimate — does wording A really beat wording B? — a fixed, well-powered split-ballot experiment is the right tool, or the bandit needs an explicit correction. If instead you want to maximise the number of good responses collected, adaptive allocation shines.
It is a cousin of, not a copy of, sequential monitoring. Optional stopping and sequential analysis decide when to stop one stream; a bandit decides which arm to feed next. And unlike computerized adaptive testing, which adapts item difficulty to measure one person precisely, a bandit adapts allocation across people to optimise a reward.

Limitations and honest caveats

The headline caveat is statistical: inference after adaptive allocation is biased, so ordinary t-tests and confidence intervals computed on bandit data are invalid, as Rafferty and colleagues demonstrated. Valid analysis needs bandit-aware estimators or a hybrid design that reserves a fraction of uniform-random allocation for unbiased estimation. Non-stationarity is the second trap: student behaviour drifts across a term, and a stationary bandit chases a moving target, potentially fixating on a spuriously good early arm. Small samples are a third: a bandit needs enough traffic to learn, and a forty-student class rarely supplies it, so the algorithm mostly explores and never earns its complexity. There is also a standardisation cost — sending different students different instruments complicates the comparability and consistency that quality assurance prizes, and can raise fairness questions if some students receive a version later judged inferior. Finally, a bandit is not a substitute for a properly powered confirmatory trial; it optimises a decision, it does not certify a scientific claim.

How Koji incorporates this

Koji's AI-moderated conversational engine is adaptive by design — it already decides which follow-up question to ask next based on what a student just said — so bandit logic fits its architecture naturally when the aim is to optimise probe selection and question wording toward richer, higher-quality responses. Koji is built to keep the exploration-versus-exploitation choice explicit rather than hidden, and to preserve a reserved fraction of uniform-random allocation so that a defensible estimate of which variant genuinely wins survives the adaptivity — directly addressing the inference problem Rafferty, Ying and Williams identified rather than ignoring it. Koji monitors the collection window for temporal drift, treating early-term and late-term responses as potentially non-stationary so an early lead does not silently freeze the design. When a team's goal is a confirmatory answer about wording rather than response quality, Koji steers them to a fixed split-ballot instead of an adaptive allocation, because the clean estimate is worth more than the marginal engagement. These mechanisms are designed to capture the participant-welfare benefit of adaptivity while mitigating its statistical cost; they do not repeal the trade-off, which is intrinsic to adaptive experimentation on any platform. The same engine powers adaptive product and customer research at koji.so, where reallocating toward a better-performing prompt raises data quality under the identical caveats.

Designing a safe adaptive experiment

A defensible adaptive study in course evaluation usually blends the two philosophies rather than choosing one. A common pattern reserves a fixed slice of respondents — say 20% — for pure uniform-random assignment across the arms and lets the bandit govern the remaining 80%: the reserved slice yields an unbiased estimate of which variant truly won, while the bandit slice delivers the participant-welfare and data-quality gains. A second safeguard is to cap how fast allocation can tilt, so a small early lead cannot starve an arm before the evidence is real. A third is to define the reward precisely before launch, because optimising for open-text richness, completion, or a specific quality score leads to different allocations, and an unstated objective quietly optimises the wrong thing. Finally, pre-register the analysis and use a bandit-aware estimator, since the honest reading of adaptively collected data is not the one an off-the-shelf t-test produces. These design choices do not remove the exploration-exploitation trade-off, but they let a team take the engagement benefit without surrendering the ability to say, defensibly, which question was actually better.

Frequently asked questions

What is a multi-armed bandit in plain terms?

It is a strategy for splitting a limited number of trials among several options when you do not know which is best, sending more trials to options that look better while still testing the others. In surveys the options are competing question wordings, probes, reminders or formats.

How is an adaptive experiment different from a normal A/B test?

A normal A/B test fixes the split (often 50/50) for the whole study and analyses it at the end. An adaptive experiment shifts the allocation toward the better-performing arm as data arrives, collecting more good responses but complicating the final statistical analysis.

Why do bandits distort statistical inference?

Because the decision to send more respondents to an arm depends on that arm's early results, the collected sample is no longer a clean random sample. Rafferty, Ying and Williams (2019) showed this inflates false-positive rates and biases effect estimates, so ordinary tests are invalid without correction.

When should I use a bandit rather than a fixed split?

Use a bandit when the goal is to maximise the quality or number of responses collected and you have enough traffic for it to learn. Use a fixed, well-powered split-ballot when the goal is a clean confirmatory estimate of which wording is better.

What is Thompson sampling?

Thompson sampling allocates each option in proportion to the posterior probability that it is the best option, drawn from a Bayesian model updated as data arrives. It dates to Thompson (1933) and is a simple, strong default for balancing exploration and exploitation.

Does adaptivity cause problems in a short teaching term?

Yes. Student behaviour changes over a term, so a bandit built for a stationary world can lock onto an arm that only looked best early on. Monitoring for temporal drift and reserving some random allocation guards against this.

References

Thompson, W. R. (1933). On the Likelihood that One Unknown Probability Exceeds Another in View of the Evidence of Two Samples. Biometrika, 25(3/4), 285-294. https://doi.org/10.1093/biomet/25.3-4.285
Robbins, H. (1952). Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society, 58(5), 527-535. https://doi.org/10.1090/S0002-9904-1952-09620-8
Scott, S. L. (2010). A modern Bayesian look at the multi-armed bandit. Applied Stochastic Models in Business and Industry, 26(6), 639-658. https://doi.org/10.1002/asmb.874
Rafferty, A. N., Ying, H., & Williams, J. J. (2019). Statistical Consequences of Using Multi-armed Bandits to Conduct Adaptive Educational Experiments. Journal of Educational Data Mining, 11(1), 47-79. https://doi.org/10.5281/zenodo.3554750
Russo, D., Van Roy, B., Kazerouni, A., Osband, I., & Wen, Z. (2018). A Tutorial on Thompson Sampling. Foundations and Trends in Machine Learning, 11(1), 1-96. https://doi.org/10.1561/2200000070

Should You Send More Students to the Better Question? Multi-Armed Bandits and Adaptive Experiments