Peer Observation vs Student Evaluations: What Each Actually Measures (and Why You Need Both)

In short: Student evaluations of teaching (SET) and peer observation reports (POR) overlap but are not interchangeable: they correlate moderately, yet each captures dimensions the other misses. Students are well placed to judge clarity, organisation and the delivered experience of a course; faculty observers are better placed to judge curriculum content, disciplinary rigour and whether learning objectives are actually targeted. Ackerman, Gross and Vigneron (2009) and Berk (2005) converge on the same conclusion: no single source is sufficient for a high-stakes judgement of teaching, and the defensible practice is triangulation across complementary sources.

The question this answers

If you already collect student ratings, do peer observations tell you anything new — or just the same thing more expensively? The validity evidence says the two are partly redundant and partly complementary, and the complementary part is exactly where high-stakes decisions go wrong when you rely on one source.

What the research says

Ackerman, Gross and Vigneron (2009), writing in the Alberta Journal of Educational Research ("Peer Observation Reports and Student Evaluations of Teaching: Who Are the Experts?"), compared peer observation reports with student evaluations of the same teaching. They found generally high correlations between students' and faculty members' overall judgements — reassuring evidence of convergent validity — but also significant differences in the weights the two groups placed on different teaching dimensions. Crucially, some dimensions valued by faculty observers, such as facilitating the achievement of key learning objectives and modelling rigorous disciplinary thinking, were largely unrecognised by students. Their conclusion was explicit: because SET and POR provide complementary information from different vantage points, it is unwise to rely on a single source as evidence of teaching effectiveness.

This sits inside a larger, well-known framework. Berk (2005), in the International Journal of Teaching and Learning in Higher Education, surveyed twelve possible sources of evidence about teaching effectiveness — student ratings, peer ratings, self-evaluation, classroom video, student interviews, alumni ratings, employer ratings, administrator ratings, teaching scholarship, teaching awards, learning-outcome measures, and teaching portfolios. His central argument is that no single source is both valid and comprehensive: student ratings are necessary and have the strongest psychometric track record of the lot, but they cannot speak to content currency or scholarly rigour, while peer observation can speak to those but is itself vulnerable to content-validity problems, several types of response bias, and weak inter-observer reliability unless the observation instrument is carefully built and observers are trained.

The classic defence of student ratings — Marsh and Roche (1997) in American Psychologist — does not contradict this. Marsh and Roche argued that well-constructed SET instruments are multidimensional, reliable, and reasonably valid against several criteria, and relatively (not entirely) free of bias. But even they framed SET as one of multiple measures of teaching effectiveness, not a sufficient one. The consistent thread across three decades is therefore not "student ratings are bad" or "peer observation is better" — it is that the two measure overlapping-but-distinct constructs, and a single source under-determines the judgement.

A useful way to state the division of expertise that emerges from this literature: students are the experts on the delivered experience — was it clear, organised, engaging, fairly assessed, responsive? Faculty observers are the experts on the designed substance — is the content current and correct, is the cognitive demand appropriate, are the stated learning objectives actually being pursued? Asking students to certify disciplinary rigour, or asking a single peer visit to certify the term-long student experience, is asking each instrument to measure outside its zone of validity.

Why it matters for course evaluation in practice

The practical failure mode is well known to anyone who sits on a promotion or quality committee: a number from a student survey gets treated as a complete verdict on teaching, and a single peer observation gets treated as either a rubber stamp or a gotcha. Both misuse the evidence.

If your institution makes summative decisions — confirmation, promotion, programme approval — on teaching, the convergent-validity evidence has three direct implications. First, collect at least two structurally different sources (typically SET plus peer observation, ideally plus a self-reflective component and some learning-outcome evidence) so that no single instrument carries a decision it was never validated to carry. Second, read divergence as signal, not noise: when students rate a course highly but a peer flags thin content, or vice versa, that disagreement is precisely the diagnostic information triangulation is supposed to surface. Third, build the peer instrument as carefully as the student one — an unstructured "drop in once and write a paragraph" observation has poor reliability and adds little, whereas a structured rubric with trained observers, as Berk insists, is what makes POR worth collecting at all.

Limitations and honest caveats

The case for triangulation is strong, but the evidence has real limits a critical reader should hold onto:

• Peer observation has its own biases. Collegiality, reciprocity ("I will rate you well if you rate me well"), and single-visit sampling error all threaten POR validity. It is not a neutral criterion against which to "check" student ratings.
• The convergent correlation is ambiguous. A moderate-to-high SET–POR correlation could reflect shared truth — or shared bias (for example, both groups responding to instructor charisma, the Dr Fox effect). Convergence is necessary for validity but not sufficient.
• Generalisability is uneven. Ackerman and colleagues studied a specific institutional context; the size of the SET–POR gap, and which dimensions diverge, vary by discipline, course level and observation protocol.
• Triangulation costs. Peer observation is labour-intensive; institutions ration it, which reintroduces sampling problems. More sources is better in principle and constrained in practice.

The honest position: triangulation reduces the risk of a single-source error; it does not deliver a bias-free composite truth.

How Koji incorporates this

Koji is an AI-native course-evaluation platform, and while it is the student-voice instrument in this picture, it is built to make that voice triangulable rather than to pretend it is the whole story — framed as designed to mitigate single-source error, not to replace peer review:

• Multidimensional, structured collection. Koji supports scale, single_choice, multiple_choice, ranking, yes_no and open_ended items, so the student instrument captures the delivery dimensions students are genuinely expert in (clarity, organisation, assessment fairness, responsiveness) rather than a single global score that invites over-reading.
• AI-moderated probing for concreteness. Where a peer observer would ask "show me evidence", Koji's AI moderator asks the student for specifics — a concrete example behind a rating — producing evidence a committee can weigh against a peer report instead of an unexplained number.
• Thematic analysis that maps to dimensions. Open-text responses are clustered into themes, making it straightforward to see which aspect of teaching students are reacting to and to line that up against the corresponding section of a peer rubric.
• Bias-aware, triangulation-ready reporting. Reports are structured to be read alongside other evidence, and Koji explicitly positions student data as one source among several rather than a standalone verdict — the same posture Berk and Ackerman recommend.

The underlying AI-moderated interview engine is the same one Koji runs for product and customer research at koji.so; in the education product it is configured to feed a multi-source quality process rather than to short-circuit it.

Building peer observation that is worth collecting

The European quality framework already assumes multi-source evidence. ESG Standard 1.5 (Teaching Staff) expects institutions to assure the competence of their teachers, and review panels increasingly distrust a teaching case built on student numbers alone. But the convergent-validity evidence carries a warning for how you operationalise the peer half: a single, unstructured, announced visit produces a report with poor inter-observer reliability and high collegiality bias — adding cost without adding signal. Berk's prescription is concrete: use a structured observation rubric tied to defined teaching dimensions, train observers on it, and where stakes are high, average across two or more observers and occasions to push inter-rater reliability into a defensible range. A worked pairing looks like this — the student instrument reports clarity, pace and assessment fairness across the whole term; two trained peers, using the same rubric on different weeks, report content currency and cognitive demand. Where the two agree, confidence is high; where they diverge — students delighted, peers concerned about thin content — the committee has precisely the diagnostic it needs, rather than a single number it must over-trust.

Related resources

• Why a Single Student Survey Cannot Stand Alone: Common-Method Bias
• What Do Student Evaluations Actually Measure? Marsh and the SEEQ
• Do Student Evaluations Measure Learning? The Uttl Meta-Analysis Revisited
• Interpreting and Reporting Student Ratings Responsibly
• Can You Fairly Rank Instructors by Their Course-Evaluation Scores?
• The Dr. Fox Effect: Does Lecturer Charisma Inflate Course Evaluations?

References

1. Ackerman, D., Gross, B. L., & Vigneron, F. (2009). Peer observation reports and student evaluations of teaching: Who are the experts? Alberta Journal of Educational Research, 55(1), 18–39. https://journalhosting.ucalgary.ca/index.php/ajer/article/view/55272 (ERIC: EJ838670)
2. Berk, R. A. (2005). Survey of 12 strategies to measure teaching effectiveness. International Journal of Teaching and Learning in Higher Education, 17(1), 48–62. https://www.isetl.org/ijtlhe/
3. Marsh, H. W., & Roche, L. A. (1997). Making students' evaluations of teaching effectiveness effective: The critical issues of validity, bias, and utility. American Psychologist, 52(11), 1187–1197. https://doi.org/10.1037/0003-066X.52.11.1187