Control for Everything Stable About an Instructor: Fixed-Effects Panel Models for Course Evaluation

In brief

If you want to know whether a change within a course or within an instructor moved the ratings — net of everything permanent about that course or teacher that you never measured — a fixed-effects (FE) panel model is the sharpest tool available. By comparing each instructor only to themselves across terms, FE silently controls for every stable confounder, measured or not: personality, discipline, campus, tenure status. The price is that FE discards all between-instructor information and cannot estimate the effect of anything that does not change over time. The Hausman test and the modern within-between formulation help you decide when FE is worth that price.

What the research says

Panel (or longitudinal) data — many instructors observed over many terms — let you separate two questions that a single cross-section confounds: why do some instructors score higher than others (between variation) and what happens when a given instructor changes something (within variation). Fixed-effects estimation isolates the second.

The mechanics are the "within transformation": you subtract each instructor's own average from every observation, so a term's rating is expressed as a deviation from that instructor's long-run mean. Anything constant for that instructor — their charisma, their subject, the difficulty of their course, their accent — is a constant and vanishes on subtraction. What remains is driven purely by within-instructor change. Allison (2009), in the standard applied treatment, frames this as FE's defining strength: it "controls for all stable characteristics of the individuals... whether or not they can be measured," which is precisely the class of confounders that plague course evaluation.

Mundlak (1978) supplied the theoretical bridge to the alternative, random-effects (RE) estimation. RE treats each instructor's effect as a random draw and, being more efficient, uses both within and between variation — but only under the strong assumption that the instructor effect is uncorrelated with the predictors. Mundlak showed that when you allow that correlation (by adding each instructor's mean predictor to an RE model), the RE slope estimate collapses exactly to the FE "within" estimate. In other words, FE is RE that refuses to trust the no-correlation assumption.

Hausman (1978) turned that comparison into a formal test. The Hausman specification test asks whether the FE and RE coefficient estimates differ by more than sampling noise; a significant difference signals that the RE assumption is violated and FE should be preferred. Bell and Jones (2015) push back on treating FE as the reflexive default: they argue that a well-specified within-between (or "REWB") multilevel model recovers the same within-estimate that FE provides and recovers the between-effect that FE throws away, while accommodating higher-level predictors FE cannot touch. Their recommendation — model both dimensions explicitly rather than discard one — is now influential across the social sciences.

Why it matters for course evaluation in practice

The central threat to any comparison of instructors is that the interesting variation is tangled with stable nuisance. Professor A teaches organic chemistry to 300 first-years at 8 a.m.; Professor B teaches a 15-person final-year seminar. Their raw averages are not comparable, and no realistic set of measured covariates fully closes the gap, because so much of what differs is unrecorded.

Fixed effects sidesteps the problem by never comparing A to B. It asks only: when Professor A adopted active learning in year three, did A's ratings move relative to A's own baseline? Because A is compared to A, the discipline, the time slot, the class size norm, and A's temperament are all held fixed automatically. For evaluating a teaching intervention, a curriculum redesign, or a policy rollout, this is exactly the estimand a quality office wants, and it is why FE underpins related designs like difference-in-differences.

FE also disciplines year-over-year narratives. A department that ranks instructors on raw means is partly ranking on immovable circumstance; an FE model reveals who actually improved. And because FE conditions out the stable instructor effect, it is a cleaner starting point than raw averages for asking which time-varying factors — a new textbook, a room change, a co-teacher — genuinely shift student experience.

Limitations and honest caveats

Fixed effects buys confounder control by spending information, and the trade-offs are real.

1. It cannot estimate time-invariant effects. If you want to know the association between an instructor's gender or discipline and their ratings, FE is silent by construction — those variables are differenced away. That makes FE the wrong tool for many bias questions (use decomposition or between-models there) even as it is the right tool for change questions.
2. It uses only "switchers." Instructors whose predictor never changes contribute nothing to the estimate. If few instructors ever adopt the intervention, FE estimates rest on a small, possibly unrepresentative subset — precise-looking but narrow.
3. It is greedy with data and can be imprecise. Discarding between variation inflates standard errors, especially with short panels (two or three terms) and small within-instructor variance.
4. It does not solve time-varying confounding. FE removes stable confounders, not shocks that coincide with the change — a pandemic, a syllabus overhaul, a cohort with unusually strong prior attainment. Those need an explicit design.
5. The FE-vs-RE choice is not automatic. As Bell and Jones (2015) argue, mechanically defaulting to FE throws away legitimate and often interesting between-instructor signal; a within-between model may serve better. The Hausman test informs the choice but should not be applied blindly.

How Koji incorporates this

A fixed-effects analysis is only as good as the panel underneath it — repeated, linkable, comparably measured observations of the same instructors and courses over time. Koji is built to produce exactly that longitudinal spine.

• A consistent, linkable measurement instrument over terms. Koji's structured question types (scale, single_choice, multiple_choice, ranking, yes_no, open_ended) keep the core instrument stable across cycles, so a term-to-term change reflects student experience rather than a reworded item. That comparability is the precondition for any within-transformation to mean something — and it connects to test equating when the instrument must evolve.
• Repeated cycles that build the panel. Because Koji supports mid-cycle and multi-term collection rather than a single end-of-term snapshot, institutions accumulate the repeated observations per instructor that FE (and within-between) models require.
• Change-focused, bias-aware reporting. Koji's reporting is designed to separate an instructor's stable baseline from movement against it, aligning with the FE estimand of "did this instructor change?" rather than "is this instructor above average?" — the more defensible basis for developmental feedback.
• Triangulation for the confounders FE cannot remove. Because FE leaves time-varying shocks untouched, Koji's open-text thematic analysis and cross-cohort triangulation help analysts spot the co-occurring event (a room move, a strike, a curriculum change) that a within-model would otherwise misattribute to the intervention. The same engine powers customer and product research at koji.so, where tracking the same accounts across quarters raises identical panel-analysis questions.

Koji supplies the clean, repeated, comparable data a fixed-effects analysis depends on; it is designed to make within-instructor comparison possible and honest, not to license causal claims the design cannot support.

Frequently asked questions

What does a fixed-effects model actually control for?

Every characteristic of an instructor (or course) that stays constant over the observed period, whether or not you measured it - discipline, personality, campus, tenure status, teaching style. It does this by comparing each instructor only to their own average across terms.

How is a fixed-effects model different from a multilevel (random-effects) model?

Random-effects models use both within- and between-instructor variation and are more efficient, but assume the instructor effect is uncorrelated with your predictors. Fixed effects drops that assumption, using only within-instructor variation. Mundlak (1978) showed the two coincide once you let RE account for that correlation.

When should I use fixed effects instead of random effects?

Use FE when you suspect stable, unmeasured instructor traits are correlated with the thing you are studying and you care about within-instructor change. A significant Hausman (1978) test points toward FE. If you also need between-instructor effects or time-invariant predictors, consider a within-between model (Bell & Jones, 2015).

Why can't fixed effects estimate the effect of gender or discipline?

Because those variables do not change within an instructor over time, the within-transformation differences them out entirely. FE is the wrong tool for time-invariant bias questions; use a decomposition or a between-model for those.

How many terms of data do I need?

FE needs repeated observations per instructor and, crucially, instructors whose predictor actually changes ("switchers"). Two terms is a bare minimum and usually underpowered; several terms with real within-instructor variation give stable estimates.

Does fixed effects prove that my teaching change caused the ratings to move?

Not on its own. FE removes stable confounders but not time-varying ones - a coincident cohort change, room move, or external shock. For causal claims, pair FE with a design such as difference-in-differences or an interrupted time series.

References

• Allison, P. D. (2009). Fixed effects regression models (Quantitative Applications in the Social Sciences, No. 160). SAGE Publications. ISBN 9780761924975.
• Mundlak, Y. (1978). On the pooling of time series and cross-section data. Econometrica, 46(1), 69-85. https://doi.org/10.2307/1913646
• Hausman, J. A. (1978). Specification tests in econometrics. Econometrica, 46(6), 1251-1271. https://doi.org/10.2307/1913827
• Bell, A., & Jones, K. (2015). Explaining fixed effects: Random effects modeling of time-series cross-sectional and panel data. Political Science Research and Methods, 3(1), 133-153. https://doi.org/10.1017/psrm.2014.7

Related resources

• Students Are Nested in Courses: Why Multilevel Models Beat Raw Averages
• Did the Teaching Change Cause the Score to Move? Difference-in-Differences for Course Evaluation
• Which Variables Should You Actually Control For? Causal Diagrams and the Backdoor Criterion
• Propensity Score Matching for Course-Evaluation Confounds
• Stop Comparing Raw Averages: Empirical-Bayes Shrinkage for Course-Evaluation Scores
• Does the Professor Students Rate Highest Teach Them Most? Value-Added Evidence vs Student Ratings