Engagement Surveys vs Course Evaluations: What NSSE Measures and Why Porter Says Be Careful

In brief

Student-engagement surveys (the National Survey of Student Engagement, NSSE, and its relatives) and course evaluations measure different things and should not be substituted for one another. Engagement surveys ask how students spend their effort — time on task, interaction with faculty, participation in enriching activities — on the theory, grounded in Astin and Kuh, that engagement drives learning. Course evaluations ask how students judge a specific course and instructor. Both are useful, and both rest on student self-report, which Porter (2011) argues fails basic validity standards: students cannot reliably recall or quantify their own behaviour, so the headline numbers may not mean what the labels claim. The defensible practice is triangulation — using engagement and evaluation data as complementary, fallible signals alongside direct evidence of learning, never as interchangeable proxies for quality.

What the research says

The conceptual case for engagement surveys is set out by Kuh (2009) in New Directions for Institutional Research. Kuh traces engagement to Astin's (1984) theory of involvement — the principle that learning rises with the time and energy students devote to educationally purposeful activities — and to the "good practices" tradition in undergraduate education. NSSE operationalises this by asking students about behaviours: how often they discuss ideas with faculty outside class, how much they collaborate with peers, how challenging the coursework is. The instrument deliberately avoids asking students to rate teaching quality directly; it asks what they did, then links those behaviours to outcomes. Kuh documents the survey's rapid adoption and its role in shifting institutional attention from inputs and reputation toward what students actually experience.

Porter (2011), writing in The Review of Higher Education, mounts the most influential critique. He examines NSSE precisely because it is the pre-eminent college-student survey — his argument being that if even NSSE fails validity tests, most student surveys do. Drawing on the cognitive-survey-methodology literature, Porter argues that NSSE items demand recall and quantification that exceed students' cognitive capacity: accurately reporting how many hours per week you studied, or how often you engaged in a behaviour over a year, is a task respondents simply cannot do reliably. He concludes that the survey fails to meet basic standards for validity and reliability and calls for a new research agenda to build genuinely valid student surveys. His broader point — that self-report is appropriate for perceptions but threatened when used to measure behaviour or learning gains — applies directly to course evaluations too.

This connects to a wider course-evaluation literature. The distinction between perceived and actual learning that Porter raises is the same one documented in the feeling-of-learning gap, and the limits of self-reported gains echo the reference-bias problem that prevents "I learned a lot" from being compared across courses. Engagement surveys and course evaluations are two windows on the student experience, each with the same underlying self-report vulnerability.

Why it matters for course evaluation in practice

For a quality-assurance office choosing what evidence to collect, three implications follow.

Engagement and evaluation answer different questions, so collecting both adds genuine information. A course can score well on satisfaction while generating little engagement, or vice versa. Engagement data can explain why an evaluation looks the way it does — low ratings in a course with high reported challenge and effort point to a very different intervention than low ratings in a disengaged course. The Course Experience Questionnaire (CEQ), which sits conceptually between the two by measuring perceived teaching approaches linked to learning, is a useful bridge.

Neither should be treated as a direct measure of teaching quality or learning. Porter's critique is a warning against the institutional habit of reading a benchmark score as if it were an objective fact about educational quality. An engagement benchmark is a noisy aggregate of self-reported behaviour; a course-evaluation mean is a noisy aggregate of satisfaction. Both inform; neither adjudicates.

Triangulation is the responsible design. The strongest quality evidence combines self-report (engagement and evaluation), behavioural traces where ethically available (attendance, participation, assessment patterns), and direct measures of learning (assessment performance, value-added analysis). This is the same multi-source logic that motivates pairing student ratings with peer observation: no single source is trustworthy alone.

A worked example

Imagine two final-year modules that both return a course-evaluation mean of 3.6 out of 5 — below the faculty average. Read alone, both look like teaching problems warranting the same intervention. Now add engagement data. In the first module, students report high challenge, frequent faculty interaction, and substantial out-of-class effort; in the second, they report low challenge, little interaction, and minimal effort. The identical evaluation score now points to opposite diagnoses: the first may be a demanding, well-taught course that students found hard and rated down for it — a pattern consistent with the feeling-of-learning gap — while the second may be a genuinely disengaging course. Acting on the evaluation mean alone would prescribe the same fix for two unlike problems. This is the practical payoff of triangulation, and also its warning: because both signals are self-reported, the engagement data is itself fallible, and the only way to break the tie with confidence is to bring in a third, non-self-report source — assessment performance, a peer observation, or a curriculum audit. Two fallible signals that agree raise confidence; when they disagree, they tell you exactly where to look harder.

Limitations and honest caveats

A careful reader should weigh several points.

Porter's critique is contested. NSSE's developers and other scholars have responded that the survey shows acceptable psychometric properties for institutional-level (not individual-level) use, and that some validity concerns are mitigated when scores are aggregated. The debate is genuine and unresolved; presenting Porter as the final word would overstate the case. The honest position is that self-report behavioural measures carry real, unsettled validity risks — not that engagement surveys are worthless.

Level of inference matters. Engagement benchmarks were designed for institution- and programme-level comparison, not for evaluating an individual course or lecturer. Pushing them down to the course level — or pushing course evaluations up to certify a whole programme — strains both instruments. Aggregation level should match the decision.

Cultural and linguistic transfer. NSSE is a North American instrument. European institutions using engagement frameworks must attend to translation equivalence and cultural differences in response style, the same issues that complicate cross-national course-evaluation comparison.

Self-report is not eliminable. Behavioural and learning measures have their own confounds (attendance does not equal engagement; assessment scores reflect prior preparation). Triangulation reduces but does not remove uncertainty; it replaces one fallible number with a more defensible, multi-signal judgement. And triangulation is only as good as the independence of its signals: if engagement items and evaluation items are answered in the same sitting, by the same self-selected respondents, in the same mood, their correlation may reflect shared method variance rather than genuine convergence — a trap we examine in common-method bias. Genuine corroboration requires at least one source that does not depend on the same students reporting on themselves at the same moment.

How Koji incorporates this

Koji is positioned as the evaluation layer in a triangulated quality system, and it is designed around the self-report caveats above rather than in denial of them.

Because Porter's central problem is that fixed-response self-report items ask students to quantify things they cannot accurately recall, Koji's AI-moderated conversational interview shifts the burden: instead of asking a student to estimate "how often" they engaged in a behaviour on a frequency scale, it can elicit concrete, episodic accounts — what the student actually did in a recent session — which cognitive-survey research treats as more reliable than global frequency estimates. This does not solve self-report's limits, but it targets the specific failure mode Porter identifies.

Koji supports structured question types (open_ended, scale, single_choice, multiple_choice, ranking, yes_no) so an institution can run engagement-style behavioural items and evaluation-style judgement items in one instrument, then read them together rather than in separate silos. Its automatic thematic analysis and triangulation across cohorts are built to combine these signals with the explicit framing that each is a fallible indicator — bias-aware reporting that flags when a conclusion rests on self-report alone and would benefit from corroboration by direct learning evidence. Koji is designed to mitigate over-interpretation, not to claim it measures learning directly.

For teams running broader engagement and experience research beyond course evaluation, Koji's core research platform at koji.so applies the same conversational-interview engine to customer and product research, where the perceived-versus-actual-behaviour problem is identical.

Related resources

• Beyond the Lecturer: What the Course Experience Questionnaire (CEQ) Measures and Why It Predicts Learning
• Students Rate the Classes Where They Learn the Most Lower: The Feeling-of-Learning Gap
• Why "I Learned a Lot" Can't Be Compared Across Courses: Reference Bias
• Peer Observation vs Student Evaluations: What Each Actually Measures (and Why You Need Both)
• Response-Shift Bias: Why Self-Reported Learning Gains in Course Evaluations Can Mislead
• Are Students Customers? The SERVQUAL Gap Model, HEdPERF, and What Service-Quality Thinking Adds to Course Evaluation

References

• Kuh, G. D. (2009). The National Survey of Student Engagement: Conceptual and empirical foundations. New Directions for Institutional Research, 2009(141), 5–20. https://doi.org/10.1002/ir.283
• Porter, S. R. (2011). Do College Student Surveys Have Any Validity? The Review of Higher Education, 35(1), 45–76. https://stephenporter.org/surveys/College%20student%20survey%20validity%20RevHE%202011.pdf
• Astin, A. W. (1984). Student involvement: A developmental theory for higher education. Journal of College Student Personnel, 25(4), 297–308.