Students Are Nested in Courses: Why Multilevel Models Beat Raw Averages for Course Evaluation

Answer first: Course-evaluation responses are not independent observations — students are clustered within sections, sections within instructors, and instructors within departments. A single raw average treats them as if they were independent, which understates uncertainty and can attribute to an instructor variance that actually belongs to the course or the cohort. Multilevel (hierarchical linear) models partition that variance into its proper levels, giving fairer instructor estimates and honest standard errors. Empirically, only a modest share of total variance in ratings sits at the section level, which is exactly why comparing raw section means is so misleading.

The structure your spreadsheet is hiding

Open any course-evaluation export and you see a long column of student ratings. It is tempting to treat each row as one independent data point and compute a mean per instructor. But the rows are not independent. Two students in the same seminar share a room, a cohort, a time slot, a syllabus, and a marking scheme; their ratings are correlated for reasons that have nothing to do with the individual student. Statisticians call this a nested or clustered (hierarchical) data structure: students (level 1) are nested within course sections (level 2), which may be nested within instructors and departments (level 3 and above).

Ignoring that structure has two costs. First, it understates uncertainty: classical formulas assume independent observations, so when observations are clustered the true standard error is larger than the naive one, and confidence intervals come out too narrow. Second, it mis-attributes variance: a flat instructor average silently mixes together differences between students, differences between sections, and differences between instructors, so a "high" or "low" score may reflect the cohort or the course rather than the teaching.

What the research says

The methodological case for multilevel modelling of student ratings is well established. Ting (2000), in Research in Higher Education, applied a multilevel perspective to a large dataset of student ratings collected in Hong Kong and showed that student-level and course-level factors operate at distinct levels and must be modelled simultaneously; treating the class mean as the only unit, or the student as an independent unit, both distort the picture. The multilevel framing also let him separate genuinely shared dimensions of teaching quality from idiosyncratic, student-specific perceptions.

The size of the levels matters enormously for interpretation. A recent multi-level regression case study of an MBA programme (Frontiers in Education, 2025) decomposed total variance and found that roughly 13% of the variance in ratings lay between course sections and about 87% within sections — i.e., between students taking the same course. The intraclass correlation (ICC) — the share of variance at the higher level — was therefore modest. That single number is decision-critical: if most of the variance is between students rather than between sections, then differences between section means are dominated by who happened to enrol and respond, not by stable differences in teaching. The same study noted that course-level variables (class size, level, elective status, cohort composition) accounted for a meaningful slice of the between-section variance — so part of what looks like an "instructor difference" is really a course-context difference.

Earlier work points the same way. Marsh & Overall (1981) found that variance attributable to the specific instructor was substantially larger than variance due to course level or course type — reassuring in that ratings do track the teacher, but only quantifiable once you model the levels separately. And a body of multilevel studies of student and instructor/course characteristics (e.g., analyses in Research in Higher Education) consistently shows that class size, discipline, and electivity exert level-2 effects that a single-level average folds invisibly into the instructor's number.

The practical upshot from this literature: multilevel models (a) produce correct standard errors by accounting for clustering, (b) let you separate instructor, course, and student variance instead of conflating them, and (c) support context-adjusted estimates that compare an instructor to what is expected given the course they actually taught. This is closely related to, but distinct from, two methods covered elsewhere on this site: generalizability theory, which uses the same variance-components logic to estimate reliability, and empirical-Bayes shrinkage, which uses the multilevel structure to pull small-sample estimates toward the mean.

Why it matters for course evaluation in practice

Raw section-mean league tables are statistically indefensible. If only ~13% of variance is between sections, then ranking instructors by their section means ranks them largely on noise and cohort composition — the very problem behind instructor misclassification in personnel decisions. A multilevel model replaces "this instructor scored 4.1, that one 4.3" with an estimate that carries an honest interval and adjusts for the course context.

Small classes are not automatically better or worse. Because the ICC tells you how much section means can be trusted, and because small sections have noisier means, a multilevel model with shrinkage protects a five-student seminar from being judged on a fragile average — directly relevant to how many responses you need for a reliable evaluation.

Context belongs in the model, not in a footnote. Known confounds such as class size and discipline can enter as level-2 predictors, so the report shows teaching signal net of course context rather than leaving readers to mentally "adjust."

Crossed effects and the team-teaching problem

A subtlety worth flagging: not all course-evaluation structures are cleanly nested. When a single student rates several instructors, or a module is team-taught, instructors and students are crossed rather than strictly nested — and the appropriate model uses crossed random effects. Forcing such data into a purely hierarchical model can misallocate variance, attributing to one instructor what actually belongs to a shared module or to a student who is systematically generous or harsh across all their ratings. Modern mixed-effects software handles crossed and nested random effects within the same framework, so the practical task is to model the design as it really is rather than as the spreadsheet's column layout implies. Getting the structure right is frequently more consequential than the choice of estimator, and it is the step most often skipped when an institution first moves beyond raw averages.

Limitations and honest caveats

Multilevel models are a sharper tool, not a magic one. They require enough higher-level units (a rough working minimum of 20–30 sections per instructor or department) to estimate level-2 variance stably; with only a handful of classes, the variance components themselves are uncertain. They assume a correctly specified structure — mis-modelling crossed effects (a student who rates several instructors, or team-taught courses) as purely nested can bias estimates. Variance partitions are sample-specific: the "13% between sections" figure from one MBA programme will not transfer unchanged to a large first-year STEM cohort. And critically, a multilevel model corrects how you analyse ratings; it cannot fix bias baked into the ratings themselves. If the underlying scores are skewed by gender, accent, or grading-leniency effects, partitioning their variance more honestly does not remove those biases — it only stops you from compounding a measurement problem with an analysis problem. Finally, the outputs are less intuitive than a mean, which raises a genuine communication burden for evaluation committees.

How Koji incorporates this

Koji for Education is designed around the recognition that a course-evaluation number is an estimate with a level structure, not a fact.

• Clustered, not flattened, data. Koji stores responses with their full hierarchy — respondent, section, instructor, course, programme — so reporting can respect the nesting rather than collapsing everything into one column. This is the precondition for any multilevel or shrinkage analysis.
• Uncertainty-first reporting. Rather than publishing a bare decimal, Koji's reporting is built to surface distributions and to flag when a section has too few responses for its mean to be trusted, mirroring the shrinkage logic that protects small classes from noisy estimates.
• Context-aware comparison. Because Koji captures course metadata (size, level, electivity, modality, cohort), comparisons can be framed against like-for-like contexts instead of a single institutional league table — the practical expression of treating class size and discipline as level-2 factors.
• Triangulation beyond the scalar. Koji's AI-moderated conversational interviews and automatic thematic analysis of open-text answers add a qualitative layer that does not depend on the variance structure of a Likert item at all, so a programme-level judgement rests on more than one fragile section mean.

These mechanisms are designed to mitigate the mis-attribution and false-precision problems the multilevel literature identifies; they do not claim to remove bias from the underlying responses. Koji's core research platform at koji.so applies the same hierarchical, uncertainty-aware thinking to product and customer research, where respondents are likewise clustered within segments and accounts.

Related Resources

• Generalizability Theory and the Reliability of Student Ratings
• Empirical-Bayes Shrinkage for Course-Evaluation Scores
• Why Small Differences in Course-Evaluation Means Are Usually Noise
• How Many Responses Do You Need for a Reliable Course Evaluation?
• Class Size and Student Evaluations: What Bedard and Kuhn Found
• Why Course-Evaluation Scores Misclassify Instructors

References

• Ting, K.-F. (2000). A Multilevel Perspective on Student Ratings of Instruction: Lessons from the Chinese Experience. Research in Higher Education, 41(5), 637–661. https://doi.org/10.1023/A:1007075516271
• Marsh, H. W., & Overall, J. U. (1981). The Relative Influence of Course Level, Course Type, and Instructor on Students' Evaluations of College Teaching. American Educational Research Journal, 18(1), 103–112. https://doi.org/10.3102/00028312018001103
• Course evaluation modeling using multi-level regression: a case study for a Master of Business Administration. (2025). Frontiers in Education. https://doi.org/10.3389/feduc.2025.1677636
• Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical Linear Models: Applications and Data Analysis Methods (2nd ed.). Sage.
• Marsh, H. W. (1987). Students' Evaluations of University Teaching: Research Findings, Methodological Issues, and Directions for Future Research. International Journal of Educational Research, 11(3), 253–388. https://doi.org/10.1016/0883-0355(87)90001-2