Did Students Even Understand the Question? Cognitive Pretesting and the Four-Stage Response Model

Most university course-evaluation forms have never been cognitively pretested — which means nobody has checked whether students interpret the questions the way the designers intended. You can have a perfectly reliable instrument, high response rates, and clean-looking dashboards, and still be aggregating answers to questions students silently rewrote in their heads. The discipline that catches this is survey methodology, and its two central tools are the four-stage response model and cognitive interviewing. Neither is exotic; both are largely absent from how universities build evaluation questionnaires.

The answer-first version: a survey response is not a readout of a fixed opinion. It is the end of a four-step cognitive process, and each step can fail quietly. Cognitive pretesting is how you find those failures before the data is collected, not after a committee puzzles over a confusing result.

What actually happens when a student answers a question

The foundational model comes from Roger Tourangeau and colleagues, developed through the 1980s and consolidated in The Psychology of Survey Response (Tourangeau, Rips & Rasinski, 2000). It breaks answering a single question into four stages:

1. Comprehension. The respondent interprets the question — its words, its intent, the reference period. "Was the workload appropriate?" invites the immediate question: appropriate for whom, compared to what, and does "workload" include the group project that was really run by one teammate?
2. Retrieval. They search memory for relevant information. Across a whole semester, what does a student actually recall about "assessment and feedback" in week four?
3. Judgment. They combine and weigh what they retrieved, and decide how much cognitive effort to spend — often not much, under end-of-term fatigue.
4. Response. They map their internal answer onto the categories you offer, which may not fit the answer they formed.

A problem at any stage produces a number that looks like data but is not measuring what you think. The reliability statistics will not tell you this; two students can be equally consistent while interpreting a question in two incompatible ways. This is why validity, not just reliability, is the harder question — and why writing "clear" questions by intuition is not enough.

Comprehension is where most evaluation questions fail

The comprehension stage is the usual culprit, and it maps onto problems this blog has covered from other angles. Double-barrelled items ("the lecturer was knowledgeable and approachable") force one answer to two questions. Vague quantifiers ("often", "adequate") mean different frequencies to different people. Undefined constructs — "effective teaching", "engaging" — invite each student to supply a private definition, which is the jingle-jangle problem lived out one respondent at a time.

For an international cohort the comprehension gap widens further: a question that reads clearly to a native speaker can carry a different pragmatic force for someone answering in a second language, a documented source of language bias. And because earlier questions prime the interpretation of later ones, comprehension is not even stable within a single form — the order in which you ask changes the answers.

None of these are exotic failures. They are ordinary, and they are invisible to anyone reading only the aggregate output.

Cognitive interviewing: the method universities skip

Cognitive interviewing is the pretesting technique built specifically to surface these failures. Gordon Willis's standard practitioner text, Cognitive Interviewing: A Tool for Improving Questionnaire Design (2005), lays out the two core probes:

• Think-aloud, where a small number of respondents verbalise their reasoning as they answer, and
• Verbal probing, where the interviewer asks targeted follow-ups: "What did 'workload' mean to you there?" "Tell me how you chose that number."

In their research synthesis on the method (Public Opinion Quarterly, 2007, 71:287–311), Paul Beatty and Gordon Willis conclude that think-aloud and verbal probing are best used together, matched to the respondents and the topic. Crucially, cognitive interviewing needs only a handful of participants — typically five to fifteen — because it is diagnosing systematic comprehension failures, not estimating a population parameter. That makes it cheap enough for any teaching-and-learning centre to run before a form goes live.

The payoff is concrete: you learn that "the course was intellectually stimulating" is read as "I enjoyed it" by some students and "it was hard" by others, and you fix the item before it contaminates a term of data — rather than after a programme committee spends an hour arguing about what a mid-range score means.

"But our questionnaire is already validated" — the counterargument

The strongest objection is that many institutions use instruments — SEEQ, IDEA, or a national template — that were developed and validated by specialists. Why re-pretest what experts already built?

Three reasons the objection does not fully hold. First, validation is context-bound. An instrument validated with North American undergraduates in the 1980s may comprehend differently for a multilingual European master's cohort in 2026; comprehension is a property of the question and its readers, not the question alone. Second, local adaptation breaks validation. The moment a committee adds three home-grown items, translates the form, or trims it "for length", the validation no longer covers what is deployed. Third, and most common, most course-evaluation forms are not validated instruments at all — they are locally assembled question sets that have never faced a single think-aloud. The counterargument, in other words, applies to a minority of the forms actually in use.

Pretesting is not a rejection of validated instruments. It is what keeps a validated instrument valid once real institutions get their hands on it. It sits alongside, not instead of, the broader total-survey-error view of where evaluation goes wrong.

Where Koji fits

Cognitive pretesting exists because a static form gives the respondent no way to say "I do not understand what you are asking" — and gives you no way to notice. The form asks; the student guesses; you never learn about the gap.

Koji's AI-moderated conversational interviews change that dynamic at the point of collection. When a student's answer is vague or seems to misread the question, the AI moderator can probe — "what do you mean by that?", "can you give an example?" — in a standardized, bias-aware way, surfacing the comprehension and retrieval problems that a paper form buries. It is not a substitute for pretesting your core scale items; it is a live, at-scale version of the verbal probing that cognitive interviewing does with fifteen people in a lab. Koji's automatic thematic analysis then aggregates what students actually meant, and its quality scoring flags responses that are thin or off-topic — the digital echo of a think-aloud that went nowhere.

Legacy SET tools inherit the comprehension gap wholesale: they can only record the guess, never interrogate it. That is the difference between a form that assumes it was understood and a conversation that checks.

The same problem — respondents quietly answering a different question than the one you asked — plagues product and customer research too. The shared AI interview engine behind koji.so applies the same probing logic outside the university, if your teams run surveys that were never pretested either.

Before your next evaluation cycle, it is worth asking a simpler question than any on the form: has anyone checked that students read it the way you meant it? See how Koji for Education builds evidence students actually understood.

FAQ

What is cognitive pretesting of a survey? Cognitive pretesting is a small-sample method — usually think-aloud interviews and verbal probing with five to fifteen respondents — used to check whether people interpret survey questions as intended before the survey is fielded. It targets comprehension and response problems that reliability statistics cannot detect.

What is the four-stage response model? Developed by Tourangeau and colleagues, it describes answering a survey question as four cognitive steps: comprehension, retrieval of relevant information, judgment, and response (mapping the answer onto the offered categories). A failure at any stage produces misleading data.

How is cognitive pretesting different from a pilot survey? A pilot tests logistics and reliability with a larger sample and produces numbers. Cognitive pretesting uses a handful of respondents to expose why a question is misread, through their verbalised reasoning — a diagnostic, not a statistical, exercise.

Do validated instruments like SEEQ still need pretesting? The instrument itself may not, but any local adaptation — added items, translation, shortening, a new student population — is no longer covered by the original validation and should be re-checked. Most locally built evaluation forms have never been pretested at all.

How does Koji help with question comprehension? Koji's AI-moderated conversational interviews can probe unclear or vague answers in real time and at scale, surfacing the comprehension and retrieval failures a static form hides. This complements, rather than replaces, cognitively pretesting your core scale items.