Response-Order Effects: Does Where an Answer Sits Change How Students Rate Your Course?

In short: Yes — the order in which answer options are presented can systematically shift course-evaluation responses, independent of what students actually think. Krosnick and Alwin (1987) showed that in self-administered, visually presented surveys the dominant bias is a primacy effect: options near the top of a list are chosen more often, and the bias is strongest among respondents who are processing the question least carefully. For a fixed-order five-point teaching scale this is usually a second-order concern next to grading-leniency or response-rate bias, but it is real, it is avoidable, and it matters most for the long nominal answer-lists ("why did you take this course", "what should change") that evaluations increasingly rely on.

What the research says

The anchor study is Jon Krosnick and Duane Alwin's "An Evaluation of a Cognitive Theory of Response-Order Effects in Survey Measurement" (Public Opinion Quarterly, 1987, 51(2), 201–219). At the time, survey methodologists had documented that the order in which response choices were offered changed which choices respondents picked, but no one had explained why. Krosnick and Alwin proposed a cognitive account and tested it with a split-ballot experiment embedded in the 1984 General Social Survey, using a variant of Kohn's parental-values task in which respondents selected the qualities they considered most important for a child, with the list presented in different orders to randomly assigned subsamples.

Their theory: respondents process options sequentially and, because deeper cognitive elaboration favours whatever is considered first, visually presented lists produce primacy effects — early options are over-selected. Crucially, the effect was concentrated among respondents low in cognitive sophistication (operationalised via education), exactly the people most likely to take shortcuts rather than weigh every option.

Two corroborating sources strengthen the picture:

• Galesic, Tourangeau, Couper and Conrad (2008), "Eye-Tracking Data: New Insights on Response Order Effects and Other Cognitive Shortcuts in Survey Responding" (Public Opinion Quarterly, 72(5), 892–913), recorded what respondents actually looked at. They found people spend more time on, and fixate more on, options near the top of a vertical list, and that those top options are then chosen more often — direct behavioural evidence for the attention-based mechanism Krosnick and Alwin inferred.
• Krosnick (1991), "Response Strategies for Coping with the Cognitive Demands of Attitude Measures in Surveys" (Applied Cognitive Psychology, 5(3), 213–236), folds response-order effects into the broader theory of satisficing: when task difficulty is high and respondent ability or motivation is low, people settle for the first acceptable answer instead of the optimal one. Response-order bias is a signature of that shortcut.

One important nuance — mode flips the direction. Visual presentation (paper, web) tends to produce primacy; auditory presentation (telephone) tends to produce recency, because the last-heard option is freshest in working memory. So you cannot assume the bias always pushes the same way.

Why it matters for course evaluation in practice

Most Student Evaluation of Teaching (SET) Likert items already have a fixed, semantically meaningful order — Strongly disagree → Strongly agree — so arbitrary primacy is muted there. The risk concentrates in three places that QA teams often overlook:

1. Long nominal answer-lists. "Which of these aspects most need improvement?" or "Why did you enrol?" with eight or twelve unordered options is the textbook condition for primacy. Items at the top get picked; items at the bottom get missed.
2. Single-choice questions with many categories, where there is no natural order to anchor attention.
3. Scale direction and numbering. Presenting "Strongly agree" first (top/left) can interact with acquiescence to nudge ratings upward. More importantly, flipping option order between cohorts manufactures artefactual change — a trend line that moves because you reordered the answers, not because teaching changed.

And the students most affected — those rushing, satisficing, completing on a phone between lectures — are precisely the ones over-represented when online response rates are low. Response-order bias and non-response bias compound.

Limitations & honest caveats

A critical reader should hold several reservations:

• Task mismatch. Krosnick and Alwin studied a values-ranking task, not a teaching Likert scale. Generalising the magnitude to a 5-point "the lecturer explained clearly" item is an extrapolation, not a measurement.
• Small effects for ordered scales. For evaluative scales with a natural order, documented response-order effects are typically small relative to the grading-leniency, workload, and identity-related biases that dominate the SET literature.
• Mode-dependence cuts both ways. Because primacy (visual) and recency (aural) point in opposite directions, "fix the order" is not a one-size prescription.
• Randomisation is not free. Randomising option order averages out primacy for nominal lists, but applying it to an evaluative scale would destroy the natural ordering and confuse respondents — so it is a targeted tool, not a universal one.
• Ageing evidence base. The foundational experiments predate mobile-first survey interfaces; scrolling, small screens, and touch targets introduce interactions that the 1987 and 2008 studies could not anticipate (see our note on device effects).

Showing you see these limits is the point: response-order effects are a genuine, well-theorised threat to some items, not a reason to distrust every number on a course report.

How Koji incorporates this

Koji is designed to mitigate — not eliminate — response-order bias through several concrete mechanisms:

• Per-respondent option randomisation for nominal items. For multiple-choice and single-choice questions without a natural order, Koji can shuffle answer options between respondents so that primacy averages out across the cohort, and it keeps the order fixed for items being trended year-on-year so comparability is protected. You get one or the other deliberately, not by accident.
• Consistent scale direction and labelling across cycles. For ordered scale questions, Koji preserves direction and verbal anchors between cohorts, so a moving trend line reflects teaching, not a reformatted instrument.
• Less reliance on long pick-lists in the first place. Koji's AI-moderated conversational interview asks open-endedly ("What, if anything, would you change about this course?") and probes follow-ups, instead of presenting a twelve-item checklist where the top option wins by position. The response is anchored to the student's own experience, then automatic thematic analysis structures the free text into comparable categories after the fact.
• Satisficing-aware quality scoring. Because response-order bias is a satisficing symptom, Koji's engagement signals — completion speed, straightlining, non-differentiation — let QA teams flag or down-weight low-effort responses rather than treat them as equivalent to careful ones (see satisficing and straightlining).

The same AI-moderated interview engine powers Koji's core research platform at koji.so for product and customer research, where long branded answer-lists create the identical primacy problem.

Related Resources

• How many scale points should a course-evaluation question have?
• Should every point on a course-evaluation scale be labelled?
• Question order and context effects
• Why students click straight down the middle: satisficing
• Do students on phones give worse course evaluations?
• Acquiescence bias and reverse-worded items

A practical decision guide for evaluation designers

If you maintain a course-evaluation instrument, response-order effects translate into a short set of design decisions you can make once and document:

1. Audit every item for list type. Separate your questions into (a) ordered evaluative scales, (b) nominal single-choice lists, and (c) nominal multi-select lists. Response-order risk lives almost entirely in (b) and (c). Ordered scales are largely protected by their natural sequence.

2. Randomise option order for nominal lists — but only those. For "which aspects need improvement" or "why did you enrol", shuffle the options per respondent so primacy averages out across the cohort. Never randomise an ordered scale; reordering Strongly disagree → Strongly agree only confuses students.

3. Freeze the order of any trended item. If you compare a question year-on-year, lock its option order. Reformatting between cohorts manufactures change that looks like a real shift in student opinion but is pure artefact — the single most common way response-order effects corrupt a QA trend line.

4. Keep nominal lists short. Primacy grows with list length because attention decays down the list. A focused six-option list collects better data than a sprawling fifteen-option one, and it lightens the cognitive load that triggers satisficing in the first place.

5. Watch the low-engagement tail. Because the bias concentrates among rushed, low-effort respondents — over-represented in low-response-rate online evaluations — pair any response-order safeguard with engagement signals (completion speed, straightlining) so you can flag the responses most likely to be position-driven rather than considered.

6. Document your choices. For accreditation, a one-paragraph note in your methodology — "nominal lists randomised; evaluative scales fixed; trended items locked" — pre-empts the reviewer question and demonstrates methodological awareness.

Treated this way, response-order bias becomes a solved, auditable design problem rather than a hidden contaminant in your reports.

References

• Krosnick, J. A., & Alwin, D. F. (1987). An evaluation of a cognitive theory of response-order effects in survey measurement. Public Opinion Quarterly, 51(2), 201–219. https://doi.org/10.1086/269029
• Galesic, M., Tourangeau, R., Couper, M. P., & Conrad, F. G. (2008). Eye-tracking data: New insights on response order effects and other cognitive shortcuts in survey responding. Public Opinion Quarterly, 72(5), 892–913. https://doi.org/10.1093/poq/nfn059
• Krosnick, J. A. (1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5(3), 213–236. https://doi.org/10.1002/acp.2350050305