The Average Effect Across Everyone, Not Inside One Classroom: GEE for Course Evaluation

In brief

Generalized estimating equations (GEE) estimate the population-average effect of a predictor when your data are clustered — students nested in courses, ratings nested in instructors — without requiring you to specify or believe a random-effects structure. GEE answers a subtly different question from a multilevel model: not "how would this change move the score within a given classroom" (the subject-specific or conditional question) but "how does the average score across the whole population of classrooms differ" (the marginal question). Its headline virtue is robustness — with the sandwich variance estimator, GEE gives valid standard errors even when your guess about the within-cluster correlation is wrong. For a quality office reporting institution-wide effects of a policy or format change, that is often exactly the estimand and exactly the safety margin you want.

What the research says

Kung-Yee Liang and Scott Zeger introduced GEE in 1986 (Biometrika 73(1):13-22, doi:10.1093/biomet/73.1.13), extending generalized linear models to correlated data. Their insight was to separate two things that maximum-likelihood mixed models bundle together: the regression coefficients you care about, and the within-cluster correlation you usually do not. GEE specifies only a working correlation structure — a guess such as "exchangeable" (every pair of responses in a course equally correlated) or "independence" — and then estimates the coefficients through an estimating equation rather than a full likelihood. The companion paper, Zeger and Liang (1986, Biometrics 42(1):121-130, doi:10.2307/2531248), worked through the binary and count cases. The remarkable property they proved: the coefficient estimates are consistent even if the working correlation is misspecified, provided the mean model is right, and the robust (sandwich) standard errors remain valid regardless. You get to be wrong about the correlation and still trust your inference.

The distinction that matters most for interpretation is marginal versus conditional. Alan Hubbard and colleagues (2010, Epidemiology 21(4):467-474, doi:10.1097/EDE.0b013e3181caeb90) — the paper wryly titled To GEE or Not to GEE — laid out the choice: a mixed (random-effects) model estimates a conditional effect, holding the cluster is latent level fixed, while GEE estimates a marginal, population-averaged effect. For a linear model these two coincide, but for nonlinear links (logistic, for count outcomes) they differ, sometimes substantially, and answer different scientific questions. Hubbard et al. argue that when the target is a population-level association — the kind of statement a policy audience wants — the marginal GEE estimate is often the more directly interpretable and less assumption-laden choice. James Hanley and colleagues (2003, American Journal of Epidemiology 157(4):364-375, doi:10.1093/aje/kwf215) provided the standard applied orientation, and Liang and Zeger is exchangeable-versus-independence guidance remains the practical default for balanced educational data.

Why it matters for course evaluation in practice

Course-evaluation data are almost always clustered, and this knowledge base already argues that ignoring the clustering — treating hundreds of student ratings as independent — understates uncertainty and inflates false positives. Multilevel models are one correct response; GEE is the other, and they are complements, not rivals. The reason to reach for GEE is when your question is genuinely population-average.

Consider three cases. A system-wide format change. A university moves mid-term feedback online and wants to report the average change in recommend rates across all courses. That is a marginal estimand; GEE targets it directly, and the sandwich errors stay valid even though the correlation of responses within a course is only roughly guessed. A binary or count outcome where interpretation matters. If the outcome is "top-boxed or not" or a comment count, the conditional log-odds from a random-effects ordinal or count model is not the population-average log-odds; a committee that reads the conditional coefficient as a population statement is simply wrong, and GEE gives them the number they think they are reading. Robustness under an uncertain correlation structure. When you have no confidence in how responses cluster — a plausible worry with cross-classified data where students span several instructors — GEE is "get the correlation wrong and still be valid" property is a genuine safeguard.

The rule of thumb is clean: if you want to describe the typical classroom is response to a change, use a mixed model; if you want the average across the population of classrooms, use GEE. Both correctly account for clustering; they simply report different, equally legitimate quantities.

Limitations and honest caveats

GEE buys robustness at real costs, and a careful reader will name them. It is not likelihood-based, so the familiar likelihood tools — AIC, likelihood-ratio tests — do not apply; model comparison uses the QIC criterion instead, and some practitioners find that unfamiliar. The sandwich estimator needs enough clusters. Its validity is asymptotic in the number of clusters, not the number of responses; with only a handful of courses or departments, sandwich standard errors are biased downward and need a small-sample correction (for example, the Mancl-DeRouen or Kauermann-Carroll adjustment). A study of ten departments cannot lean on the naive sandwich. Missing data assumptions are stricter. Standard GEE is consistent under missing-completely-at-random; if nonresponse is related to the outcome — the norm in course evaluation, where dissatisfied students may skip — you need weighted GEE or a different approach, which connects to the broader nonresponse problem. It gives no cluster-level predictions. Because there is no random effect, GEE cannot produce a shrunken estimate for a specific instructor the way empirical-Bayes or multilevel methods can; if the goal is to rank or report individual instructors, GEE is the wrong tool. And it models the outcome you supply, biases included — a marginal effect on a biased score is a marginal effect on a biased score.

How Koji incorporates this

Koji is analytics are designed to distinguish the two questions that clustered evaluation data can answer, rather than collapsing them into one ambiguous coefficient. When an institution asks about the system-wide effect of a change — a new evaluation format, a timing shift, a curriculum-wide intervention — Koji is reporting is built to estimate a population-average effect with cluster-robust standard errors, so the headline number is the marginal quantity a policy audience actually intends, and the uncertainty around it does not pretend hundreds of clustered responses were independent. Because the robust variance is only trustworthy when there are enough clusters, Koji is designed to surface how many courses or departments underlie an estimate and to flag when that count is too small for the naive sandwich, steering the analysis toward a small-sample correction rather than a falsely tight interval. For questions that are genuinely about a specific instructor or course, Koji leans on its shrinkage and multilevel reporting instead, keeping the marginal and conditional estimands clearly labelled so a committee never reads a population-average trend as a within-classroom effect or the reverse. And because response-driven missingness threatens the GEE assumption most directly, Koji pairs marginal modelling with its representativeness and nonresponse diagnostics. Koji is core research platform at koji.so applies the same marginal-versus-conditional discipline to product and customer research, where population-average effects of a change are frequently the reporting target.

The disciplined message for an analyst is that clustering is not only a nuisance to be corrected but a fork in the question itself. GEE is the tool for the population-average branch, and its robustness to a mis-guessed correlation structure is precisely why it belongs in the reporting toolkit next to, not instead of, multilevel models.

Frequently asked questions

How is GEE different from a multilevel or mixed model?

Both handle clustered data, but they estimate different things. A mixed model gives a conditional (subject-specific) effect — the change within a given classroom. GEE gives a marginal (population-average) effect — the change in the average across all classrooms. For linear models these coincide; for logistic or count outcomes they differ and answer different questions.

What is the "working correlation structure" and does getting it wrong matter?

It is your guess about how responses within a cluster correlate (for example, exchangeable or independence). GEE is coefficient estimates stay consistent and the robust standard errors stay valid even if that guess is wrong, as long as the mean model is correct. A good guess improves efficiency but is not required for validity.

When should I not use GEE?

When you want to describe or rank individual clusters (a specific instructor), because GEE produces no cluster-level estimates; when you have very few clusters, because the sandwich standard errors are biased without a small-sample correction; and when missingness depends on the outcome, unless you use weighted GEE.

Does GEE need a lot of data?

It needs enough clusters, not just enough responses. Its robust variance is justified asymptotically in the number of courses or departments; a handful of large clusters is not the same as many clusters and calls for a correction.

Why would a committee prefer the marginal estimate?

Because policy statements are usually about populations — "recommend rates rose on average across courses" — not about a hypothetical fixed classroom. The marginal GEE estimate is the number that matches that sentence; the conditional coefficient from a mixed model does not, especially for binary or count outcomes.

Can GEE handle non-normal outcomes like counts or top-box indicators?

Yes. Like generalized linear models, GEE supports logistic links for binary/top-box outcomes and log links for counts, while accounting for the within-cluster correlation — that flexibility is much of its appeal.

References

• Liang, K.-Y., & Zeger, S. L. (1986). Longitudinal data analysis using generalized linear models. Biometrika, 73(1), 13-22. doi:10.1093/biomet/73.1.13
• Zeger, S. L., & Liang, K.-Y. (1986). Longitudinal data analysis for discrete and continuous outcomes. Biometrics, 42(1), 121-130. doi:10.2307/2531248
• Hanley, J. A., Negassa, A., Edwardes, M. D., & Forrester, J. E. (2003). Statistical analysis of correlated data using generalized estimating equations: an orientation. American Journal of Epidemiology, 157(4), 364-375. doi:10.1093/aje/kwf215
• Hubbard, A. E., Ahern, J., Fleischer, N. L., Van der Laan, M., Lippman, S. A., Jewell, N., Bruckner, T., & Satariano, W. A. (2010). To GEE or not to GEE: comparing population average and mixed models for estimating the associations between neighborhood risk factors and health. Epidemiology, 21(4), 467-474. doi:10.1097/EDE.0b013e3181caeb90

Related Resources

• Students Are Nested in Courses: Why Multilevel Models Beat Raw Averages for Course Evaluation
• One Student, Many Teachers: Cross-Classified and Multiple-Membership Models for Fair Course Evaluation
• Ordinal Regression for Course-Evaluation Data: Why Cumulative-Link Models Beat Averaging Likert Scores
• Count Models for Course Evaluation: Why You Should Not Average the Number of Comments
• Stop Comparing Raw Averages: Empirical-Bayes Shrinkage for Course-Evaluation Scores
• Can You Weight Your Way Out of a Low Response Rate? Post-Stratification and Nonresponse Adjustment