Is One "Overall" Question Enough? Single-Item vs Multi-Item Measures in Course Evaluation

In short: A single well-written global item (for example, "Overall, how would you rate this course?") can be a defensible measure when the construct it targets is narrow and concrete — and the empirical evidence is more forgiving than psychometric folklore suggests. Gogol et al. (2014), testing single-item versions of motivational constructs against full scales in a sample of nearly 3,900 students, found single items correlated .88–.97 with their multi-item parents. But single items cannot estimate their own reliability, cannot capture a multidimensional construct like "teaching effectiveness", and are fragile to a bad question. The practical answer for course evaluation is not "one or many" but "the right number for the job": a short multidimensional core for diagnosis, plus open conversation for the why.

What the research says

The anchor is Katarzyna Gogol and colleagues' "'My Questionnaire is Too Long!' The assessments of motivational-affective constructs with three-item and single-item measures" (Contemporary Educational Psychology, 2014, 39(3), 188–205). Their motivation is one every QA officer recognises: testing time is scarce, students fatigue, and long batteries depress response quality and rates. They compared three-item short scales and single-item measures against established long scales for academic anxiety and academic self-concept, using data from 3,879 ninth-grade students.

The findings were strikingly reassuring for brevity. The short three-item forms showed satisfactory reliability (α ≈ .75–.89) and very high correlations with the long scales (r ≈ .88–.97). Single items performed less well than three-item scales but still carried substantial valid signal for these reasonably narrow, concrete constructs. The headline lesson: shortening a scale costs far less measurement quality than the "longer is always better" reflex assumes — provided the construct is well-defined.

Two corroborating sources frame the boundaries:

• Wanous, Reichers and Hudy (1997), "Overall Job Satisfaction: How Good Are Single-Item Measures?" (Journal of Applied Psychology, 82(2), 247–252), meta-analysed single- vs multi-item measures of overall job satisfaction and estimated the single-item measure's reliability via the correction for attenuation at a respectable ~.67 minimum. Their conclusion — single items are acceptable for global, concrete constructs — is the most-cited licence for one-item global ratings, and "overall satisfaction with a course" is a close analogue.
• Allen, Iliescu and Greiff (2022), the editorial "Single Item Measures in Psychological Science: A Call to Action" (European Journal of Psychological Assessment, 38(1), 1–5), argues the discipline has been needlessly dogmatic: single items are appropriate when the construct is unidimensional and unambiguous, when respondent burden is a real threat, and when face validity is high — but inappropriate for broad, multifaceted constructs where you need to know which facet is failing.

The synthesis across these is consistent: the number of items should follow the breadth of the construct, not a blanket rule.

Why it matters for course evaluation in practice

"Teaching effectiveness" is the textbook multidimensional construct. Marsh's decades of work on the SEEQ (see our multidimensionality note) shows it decomposes into learning/value, enthusiasm, organisation, group interaction, individual rapport, breadth, assessment, workload, and more. A single "rate this course" item collapses all of that into one number — useful as a headline, useless as a diagnosis. If the score drops, one item cannot tell you whether the problem is assessment clarity or pacing.

This produces three practical rules:

1. A single global item is fine as a summary KPI — it correlates strongly with the underlying experience and is cheap to trend. It is the course-evaluation analogue of "overall satisfaction", exactly the case Wanous et al. endorse.
2. A single global item is not a diagnosis. For improvement and for accreditation evidence (which expects you to show you can locate and act on specific weaknesses), you need a small multidimensional core.
3. Short beats long. Gogol et al. is a direct argument against the 30-item monster questionnaire: three good items per dimension recover almost all the signal, while protecting the response rate and reducing the satisficing that long surveys induce (see questionnaire length).

Limitations & honest caveats

A careful reader should resist over-reading the brevity result:

• Construct breadth is the hidden moderator. Gogol et al. tested narrow constructs (anxiety, self-concept). "Overall teaching quality" is broader, so single-item validity is plausibly lower than their figures imply. Wanous et al.'s ~.67 is more honest for a global teaching item than the .9+ correlations seen for narrow scales.
• Single items cannot report their own reliability. Internal-consistency reliability (Cronbach's alpha) requires multiple items. With one item you must estimate reliability indirectly (test–retest, correction for attenuation) and accept more uncertainty (see Cronbach's alpha).
• One item is fragile. If the single question is double-barrelled, ambiguous, or hits a bad word for some subgroup, there is no second item to dilute the damage. Multi-item scales are partly robust because of redundancy.
• Sample specificity. Gogol et al. used ninth-graders in one schooling system; university adults answering teaching evaluations differ in motivation and reading. Direct transfer of effect sizes is an assumption, not a finding.
• Aggregation still needs enough responses. Brevity raises per-student data quality, but the reliability of the course mean still depends on how many students respond (see how many responses you need).

How Koji incorporates this

Koji is built around exactly the "right number for the job" principle, and is designed to mitigate the single-item/long-battery trade-off rather than pick a losing side of it:

• A short multidimensional core, not a 30-item slog. Koji supports structured question types — scale, single_choice, multiple_choice, ranking, yes_no, and open_ended — so you can run a compact, well-targeted set of dimension items (organisation, assessment clarity, workload, value) plus a single global KPI, in line with the Gogol et al. evidence that short forms recover most of the signal.
• Depth without item inflation. Where a traditional survey would bolt on more Likert items to chase nuance, Koji's AI-moderated conversational interview probes the why behind the global rating in the respondent's own words. You get the brevity of a single headline number and diagnostic richness — without lengthening the fixed instrument and triggering fatigue.
• Automatic thematic analysis turns that open text into structured, dimension-level themes, effectively reconstructing the multidimensional picture a single Likert item cannot give — then maps it back to the global score so a falling KPI comes with an explanation, not just an alarm.
• Quality scoring flags low-effort responses, which matters more when you rely on few items, because there is less redundancy to absorb a careless answer.

Koji never claims a single number captures teaching effectiveness; it is designed to pair a concise quantitative core with conversational depth. The same engine runs product and customer research on Koji's core platform at koji.so, where the identical "one NPS number vs real diagnosis" tension shows up.

Related Resources

• What do student evaluations actually measure? Marsh and multidimensionality
• How long should a course evaluation be?
• Cronbach's alpha and course-evaluation reliability
• How many responses do you need for a reliable course evaluation?
• Why students click straight down the middle: satisficing
• Optimal number of rating-scale points

A worked example: trimming a bloated instrument

Consider a faculty whose end-of-module survey has grown to 32 Likert items over a decade of well-meaning additions. Response rates have slid below 25%, and open comments complain the survey is "endless". The single-item evidence offers a disciplined way to cut without losing diagnostic power.

Step 1 — Name the dimensions you actually act on. In practice, most committees act on a handful: organisation and clarity, assessment fairness, workload, intellectual challenge, and overall value. Constructs nobody ever uses in a decision are candidates for deletion regardless of their psychometrics.

Step 2 — Keep two to three items per retained dimension. Gogol et al.'s finding that three-item short forms recover .88–.97 of the long-scale signal is the licence to drop the fourth, fifth, and sixth item measuring the same thing. Three good items per dimension is enough for a reliable, factor-analysable subscale.

Step 3 — Add one global KPI. A single "Overall, how would you rate this course?" item gives you the cheap, trendable headline number — the case Wanous et al. (1997) specifically endorse for global, concrete constructs — without pretending it diagnoses anything.

Step 4 — Move the nuance to conversation, not more items. Instead of bolting on yet more Likert statements to capture edge cases, ask one or two open, probing questions. This is where a conversational instrument earns its keep: depth comes from follow-up dialogue, not item count.

A 32-item battery becomes roughly 12 structured items plus a global rating and a short conversational tail. The reliability of each retained subscale is essentially preserved, the response rate recovers as burden falls, and — critically — the survey now distinguishes which dimension moved when the overall number changes. Shortening is not a downgrade here; done with the construct-breadth logic above, it is a measurement upgrade that also protects your sample.

References

• Gogol, K., Brunner, M., Goetz, T., Martin, R., Ugen, S., Keller, U., Fischbach, A., & Preckel, F. (2014). "My Questionnaire is Too Long!" The assessments of motivational-affective constructs with three-item and single-item measures. Contemporary Educational Psychology, 39(3), 188–205. https://doi.org/10.1016/j.cedpsych.2014.04.002
• Wanous, J. P., Reichers, A. E., & Hudy, M. J. (1997). Overall job satisfaction: How good are single-item measures? Journal of Applied Psychology, 82(2), 247–252. https://doi.org/10.1037/0021-9010.82.2.247
• Allen, M. S., Iliescu, D., & Greiff, S. (2022). Single item measures in psychological science: A call to action. European Journal of Psychological Assessment, 38(1), 1–5. https://doi.org/10.1027/1015-5759/a000699