Computerized Adaptive Testing for Shorter Course-Evaluation Surveys

In brief

Computerized adaptive testing (CAT) selects each question dynamically: after a student answers, the system estimates where they sit on the trait being measured and picks the next item that is most informative at that point, stopping when a target precision is reached. Because a well-matched item carries far more information than a generic one, CAT can match the reliability of a fixed 30-item scale with 10–15 items (Weiss 1982). The prize for course evaluation is obvious — shorter surveys, less fatigue, higher completion. The price is equally real: CAT requires a large item bank whose questions have been calibrated under item response theory (IRT) and that measure essentially one dimension. Most evaluation instruments meet none of those preconditions, so CAT is best understood as a direction of travel and a design principle, not a plug-in.

What the research says

Weiss (1982, Applied Psychological Measurement 6(4):473–492) laid out the case that started operational adaptive testing: by tailoring item difficulty to each examinee, adaptive tests achieve equal or better measurement precision with substantially fewer items than fixed forms, and they measure equally well across the whole trait range instead of only in the middle. The machinery is IRT: each item has calibrated parameters (difficulty, discrimination), the system maintains a running estimate of the respondent's latent trait, and it chooses the next item to maximise information at the current estimate, updating after every response until the standard error falls below a threshold or a length cap is hit.

The canonical reference collections — Wainer et al.'s Computerized Adaptive Testing: A Primer (2000) and van der Linden and Glas's Elements of Adaptive Testing (2010) — codify the four moving parts every CAT needs: a calibrated item bank, a starting rule, an item-selection rule (typically maximum Fisher information, often with randomisation for exposure control), and a stopping rule (fixed length, or fixed precision). They also document the failure modes: item-exposure (the most informative items get over-used and leak), content balancing (pure information-maximising selection can skip whole content areas), and the dependence of everything on the quality of the initial calibration.

Practically, adaptive designs are now routine outside cognitive testing. Large patient-reported-outcome systems administer attitude and symptom scales adaptively, demonstrating that CAT extends beyond ability tests to the kind of self-report, opinion-style measurement course evaluation actually involves — provided the scale is unidimensional and well-calibrated. Open tooling has lowered the barrier to simulation: Magis and Raîche (2012, Journal of Statistical Software 48(8):1–31) released the R package catR, which lets a QA team simulate an adaptive version of an existing item bank — starting rule, selection rule, stopping rule — and measure how many items it would actually save before committing to live adaptive delivery.

Why it matters for course evaluation in practice

The chronic disease of course evaluation is length. Longer instruments drive breakoff and satisficing, and the standard fix — cutting items — throws away coverage. CAT offers a third path: keep a large bank of items covering every facet of teaching, but show each student only the handful that are most informative for them, stopping once you know their standing precisely enough. A student whose first three answers are uniformly positive does not need the remaining fifteen confirmatory items; a student giving mixed signals gets more items exactly where the uncertainty is. The result is a short survey for most respondents and precise measurement for all — the opposite of the one-size-fits-all fixed form.

There is a second, subtler benefit. Fixed evaluations measure best near the average and poorly at the extremes, which is where consequential cases live (the struggling course, the outstanding one). Because CAT targets information to each respondent's location, it measures the tails as precisely as the middle, giving fairer estimates for exactly the courses a quality committee most needs to judge. Even if a department never runs a true CAT, the simulation exercise — calibrate the bank, ask how many items an adaptive version would need — reveals which items carry information and which are redundant, informing a better fixed short form.

Limitations and honest caveats

CAT is demanding, and overselling it would mislead. It needs a big, calibrated bank. IRT calibration wants hundreds of responses per item and a bank several times larger than any test length; a 12-item evaluation cannot be made adaptive. It assumes unidimensionality. Teaching quality is multidimensional — clarity, feedback, workload, inclusion are distinct — so a single adaptive scale either collapses them (losing the multidimensional signal) or requires a multidimensional CAT, which is far harder. Opinion is not ability. IRT models were built for right/wrong items; applying them to attitudes assumes a stable latent trait and monotone item responses that response styles and acquiescence can violate. Comparability and transparency suffer — different students see different items, which complicates item-level reporting and can feel opaque or unfair to faculty who expect everyone answered the same questions. And exposure/security control adds engineering that only pays off at scale. For most institutions, planned missingness — randomly assigning each student a subset of items — delivers much of the length saving with none of the IRT overhead, and is the pragmatic first step.

How Koji incorporates this

Koji does not claim to run a psychometric CAT, and it should not — most course-evaluation banks are neither large enough nor calibrated. What Koji implements is the underlying principle: adaptive next-question selection. Its AI-moderated conversational interview branches on what a student has already said, skipping items rendered redundant by earlier answers and probing deeper exactly where uncertainty or emotion appears — the same "spend questions where they are informative" logic CAT formalises, applied to a mix of structured items (scale, single_choice, ranking, yes_no) and open-ended follow-ups rather than to a single IRT dimension. For institutions that do build a calibrated bank, Koji's structured-question engine can hold the item parameters and support IRT-based selection and stopping rules, with content-balancing and exposure controls so the adaptive path still covers every required facet of teaching. Koji frames adaptivity as "designed to shorten the survey while preserving coverage", not as a precision guarantee, and it logs which items each respondent saw so reporting stays honest about differential administration. The same adaptive-interview engine powers Koji's core research platform at koji.so, where shortening a customer survey without losing signal is the identical problem.

Frequently asked questions

How much shorter can an adaptive course evaluation be?

Simulations on well-calibrated banks typically show adaptive tests reaching the reliability of a 30-item fixed form with 10–15 items, though the exact saving depends on how informative and well-spread your items are. Run a catR simulation on your own bank before promising a number.

Can CAT be used for opinions and attitudes, not just ability tests?

Yes, adaptive administration is now standard for self-report symptom and attitude scales, but only when the scale is unidimensional and the items are IRT-calibrated. Multidimensional teaching-quality constructs need multidimensional CAT, which is considerably harder.

What is the minimum item bank size for a course-evaluation CAT?

There is no single number, but you generally need far more items than any test length and hundreds of responses per item to calibrate them. A short fixed evaluation cannot simply be switched to adaptive mode.

Is CAT the same as branching or skip logic?

No. Skip logic follows fixed author-written rules ("if online, ask about the LMS"). CAT selects the next item statistically, to maximise measurement information given the running trait estimate. Koji's conversational branching sits between the two.

Does adaptive testing make comparisons between students unfair?

Different students see different items, so item-level comparison needs care, but IRT places every respondent on the same latent scale, so the trait estimates are comparable by design. The trade-off is transparency, which you manage by logging and disclosing which items each person saw.

What is a simpler alternative if we cannot calibrate an item bank?

Planned missingness — randomly assigning each student a subset of items from a longer instrument — captures much of the length saving without IRT, and matrix or split-ballot designs let you cover a large item pool across the cohort rather than per student.

Related Resources

References

Weiss, D. J. (1982). Improving measurement quality and efficiency with adaptive testing. Applied Psychological Measurement, 6(4), 473–492. https://doi.org/10.1177/014662168200600408
Wainer, H., Dorans, N. J., Flaugher, R., Green, B. F., & Mislevy, R. J. (2000). Computerized Adaptive Testing: A Primer (2nd ed.). Lawrence Erlbaum. ISBN 978-0805835113.
van der Linden, W. J., & Glas, C. A. W. (Eds.). (2010). Elements of Adaptive Testing. Springer. https://doi.org/10.1007/978-0-387-85461-8
Magis, D., & Raîche, G. (2012). Random generation of response patterns under computerized adaptive testing with the R package catR. Journal of Statistical Software, 48(8), 1–31. https://doi.org/10.18637/jss.v048.i08

Ask Fewer Questions, Measure More Precisely: Computerized Adaptive Testing for Course Evaluations