How Long Should a Course Evaluation Be? Questionnaire Length, Breakoff, and Answer Quality

In brief: Longer course evaluations reduce the share of students who start and finish them, and the questions placed late in a long instrument receive faster, shorter, more skipped, and less differentiated answers — Galesic and Bosnjak (2009) demonstrated all of these effects experimentally. But the relationship between length and response rate is weaker than most administrators assume (Rolstad, Adler & Rydén, 2011), so the right design goal is not "as short as possible" but "every item earns its place." Cut redundant Likert grids, not the open-text questions that carry the most diagnostic value.

The question every evaluation designer eventually asks

Add one more question and you learn one more thing — but you also nudge a few more students to abandon the survey, and you quietly erode the quality of the answers to everything that follows. Course-evaluation instruments accrete over years: a department adds an item about the virtual learning environment, a dean wants a question on employability, the library asks for one line on resources. Nobody removes anything. The result is a 40-item instrument that takes twelve minutes and that students increasingly click through on autopilot. How long is too long, and what does length actually cost you?

What the research says

The cleanest experimental evidence comes from Galesic and Bosnjak (2009), published in Public Opinion Quarterly. They manipulated the stated length of a web questionnaire (telling respondents it would take 10, 20, or 30 minutes) and randomised the position of thematically grouped question blocks. Two findings matter for course evaluation. First, the longer the announced length, the fewer people started and the fewer completed — perceived burden suppresses participation before a single question is answered. Second, and more subtly, questions placed later in the instrument were answered worse regardless of their content: respondents spent less time on them, left more of them blank, wrote shorter open-text answers, and showed less variability across the items of a rating grid (more straightlining). Because the blocks were randomised, this degradation is a pure position effect — the same question yields poorer data simply by appearing on page nine instead of page two.

This connects to a broader survey-methodology literature on respondent burden. Crawford, Couper and Lamias (2001), in a web survey of over 4,500 university students, showed that perceived burden — shaped by how long the survey is announced to be and whether a progress indicator is shown — directly affects whether students participate and where they break off. Burden is partly psychological: a progress bar that advances slowly can increase abandonment relative to no bar at all.

The counterweight, and the reason "shorter is always better" is too glib, is Rolstad, Adler and Rydén (2011) in Value in Health. Their systematic review and meta-analysis of 20 studies relating questionnaire length to response rate found the evidence for a strong length-response penalty surprisingly weak. Length clearly can depress participation, but in many field studies the effect is small relative to salience, relationship with the sender, and topic interest. The practical reading is not "length does not matter" but "length is one lever among several, and a relevant, well-introduced survey from a trusted sender can sustain participation even at moderate length."

Putting the three together: length imposes two distinct costs that must not be conflated. One is a participation cost (fewer students start/finish), which is real but smaller and more context-dependent than folklore suggests. The other is a measurement cost (the answers you do collect get worse toward the end), which Galesic and Bosnjak show is robust and position-driven. The second cost is the one course-evaluation designers systematically underestimate.

Why it matters for course evaluation in practice

Three implications follow directly.

1. Put your highest-value questions first. If late items get worse answers, the single open-ended question whose verbatim comments your teaching staff actually read should not be buried at the bottom after fifteen Likert items. Position is a design variable, not an afterthought. The conventional layout — demographics and global ratings first, open text last — is almost exactly backwards if open text is where your diagnostic value lives.

2. Audit for redundancy, not just count. A 30-item instrument in which five items load on the same underlying factor is worse than a 20-item instrument with no redundancy: it costs you completion and answer quality while measuring nothing extra. The goal is information per minute of student time, not minimum length for its own sake.

3. Beware false economies. Cutting open-text prompts to "save time" is often the wrong cut. Rich qualitative feedback is the most actionable output of an evaluation; a closed Likert grid is cheap to answer but frequently straightlined. Length reduction should target low-information closed items, not the few questions that change what teachers do next term. This is also why raw item count is a poor proxy for burden — three thoughtful open questions can feel lighter and yield more than twenty grid rows.

For quality-assurance officers, this reframes a perennial committee fight. The argument is rarely "should the survey be shorter?" in the abstract; it is "which stakeholder's pet item gets cut?" The evidence gives you a principled answer: keep items that (a) measure something no other item measures and (b) feed a decision someone will actually make. Everything else is burden.

Limitations and honest caveats

Several caveats temper any direct transfer of this evidence to a specific institution.

• Generalisability of populations. Galesic and Bosnjak studied an online access panel, not enrolled students completing an end-of-term evaluation. Students are a captive, repeatedly-surveyed population with a different motivational profile (and different incentives) than panellists. Position effects are likely to generalise because they reflect cognitive fatigue, but the absolute magnitudes will not transfer one-to-one.
• Stated vs experienced length. The participation effect Galesic and Bosnjak isolate is driven by announced length. In many course-evaluation systems students are not told a duration up front, so the front-end deterrent may operate through other cues (number of pages visible, scroll length) rather than a stated minute count.
• Length is confounded with content. Longer surveys often are more tedious — more grids, more repetition. Disentangling "length per se" from "the kind of content that makes surveys long" is genuinely hard, which is part of why the Rolstad meta-analysis finds heterogeneous results.
• Breakoff is not always informative loss. A student who abandons after the items they cared about may have given you the data that mattered; non-completion is not equivalent to missing the important signal. Completion-rate maximisation can be the wrong target.
• Replication and effect sizes. As with much survey-methods work, individual experiments report modest effect sizes and the field has not converged on a universal length threshold. Treat "keep it lean and front-load value" as a robust heuristic, not a precise rule with a magic number of items.

A PhD reader should leave this section reassured that we are not claiming a deterministic law — we are claiming a well-replicated direction of effect (later and longer is worse, all else equal) whose magnitude is institution-specific.

How Koji incorporates this

Koji is an AI-native, AI-moderated course-evaluation platform, and several of its mechanisms are designed to address precisely the length-versus-quality tension the research identifies — without resolving it by simply deleting questions (which would sacrifice coverage).

• Conversational, adaptive sequencing instead of a fixed grid. Rather than presenting every student the same long static form, Koji runs an AI-moderated conversational interview that can branch. Students who have little to say on a topic move on quickly; students with something specific are probed further. This concentrates respondent effort where it is informative and reduces the dead weight of irrelevant items — a structural answer to "information per minute" rather than to "minimum item count."
• Front-loading high-value, open-ended probing. Because Koji can elicit and then thematically analyse open-text responses at scale, the platform is designed to lead with conversational, open_ended questions that carry diagnostic value, rather than relegating them to the end where Galesic and Bosnjak show answers degrade. The AI follow-up turns a single prompt into the depth that a long battery of closed items was trying — and failing — to capture.
• Mitigating late-item fatigue. Adaptive flow shortens the experienced path for most students, reducing the cumulative fatigue that produces straightlining and item non-response toward the end of long instruments.
• Structured question types used deliberately. Koji supports open_ended, scale, single_choice, multiple_choice, ranking, and yes_no items, so designers can match each question to the lightest format that captures the construct — and the platform's quality scoring flags low-effort or non-substantive responses so a short, attentive answer is not mistaken for a long, careless one.

The aim is not to claim Koji eliminates response burden — no instrument does. It is to replace a static, accreting, fatigue-inducing form with an adaptive conversation that spends each student's limited attention on the questions that will actually change teaching. Koji is built for European higher education and its quality-assurance frameworks; the same AI-moderated interview engine also powers product and customer research on the core platform at koji.so, where survey-length and engagement trade-offs are equally acute.

Related Resources

• Survey Fatigue: Why Over-Surveying Students Quietly Wrecks Your Response Rates
• Why Students Click Straight Down the Middle: Satisficing in Course Evaluations
• What Actually Raises Course-Evaluation Response Rates? The Experimental Evidence
• Online vs. Paper Course Evaluations: Do Lower Response Rates Mean Worse Data?
• How Many Responses Do You Need for a Reliable Course Evaluation?
• How Many Scale Points Should a Course-Evaluation Question Have?

References

• Galesic, M., & Bosnjak, M. (2009). Effects of Questionnaire Length on Participation and Indicators of Response Quality in a Web Survey. Public Opinion Quarterly, 73(2), 349–360. https://doi.org/10.1093/poq/nfp031
• Crawford, S. D., Couper, M. P., & Lamias, M. J. (2001). Web Surveys: Perceptions of Burden. Social Science Computer Review, 19(2), 146–162. https://doi.org/10.1177/089443930101900202
• Rolstad, S., Adler, J., & Rydén, A. (2011). Response Burden and Questionnaire Length: Is Shorter Better? A Review and Meta-analysis. Value in Health, 14(8), 1101–1108. https://doi.org/10.1016/j.jval.2011.06.003