Exactly Balanced Comparison Groups Without the Guesswork: Entropy Balancing for Course Evaluation

In brief

When you compare course-evaluation scores between two groups — online versus in-person, a redesigned course versus its old version, one instructor versus a peer — the groups differ on more than the thing you care about. Confounders such as class size, discipline, and student ability contaminate the comparison. Propensity-score matching is the familiar fix, but it discards unmatched cases and forces you to iterate a propensity model until the matched groups happen to balance. Entropy balancing, introduced by Hainmueller (2012), takes a different route: it directly reweights the comparison group so that its weighted means (and, if you ask, variances and higher moments) exactly equal the treatment group's, in a single optimisation, while staying as close as possible to uniform weights so information is retained.

The result is exact balance on the covariates you specify with no iterative search and no case-dropping — properties that make it attractive for the small, messy datasets a quality office actually has. It sits alongside propensity-score matching and the E-value as part of a defensible confounding-adjustment toolkit, not a replacement for careful causal thinking.

What the research says

Hainmueller (2012), Entropy Balancing for Causal Effects: A Multivariate Reweighting Method to Produce Balanced Samples in Observational Studies (Political Analysis, 20(1), 25-46), which won the Society for Political Methodology's Miller Prize, sets out the method. The problem it solves is a well-known weakness of propensity-score approaches: you estimate a propensity model, weight or match on it, then check whether covariates balanced; if they did not, you re-specify the model and try again. Balance is an outcome you hope for, achieved by trial and error.

Entropy balancing inverts this. It treats balance as a constraint and solves for the set of unit weights that:

1. satisfy the balance constraints exactly — the reweighted comparison group matches the treatment group on the specified moments (means, and optionally variances and skewness) of every covariate; and
2. deviate as little as possible from uniform base weights, measured by an entropy (Kullback-Leibler) distance, so the reweighting does not lean on a handful of cases.

Because balance is imposed rather than hoped for, there is no iterative re-specification. The weights are then used in a standard weighted regression or weighted mean comparison. Hainmueller and Xu (2013), ebalance: A Stata Package for Entropy Balancing (Journal of Statistical Software, 54(7)), provide the implementation that made the method routine, and equivalents now exist in R and Python.

The method has an important theoretical property. Zhao and Percival (2017), Entropy Balancing is Doubly Robust (Journal of Causal Inference, 5(1)), proved that entropy balancing is doubly robust: the estimated treatment effect is consistent if either the outcome model or the (implicit) propensity model is correctly specified — you do not need both right. This is a genuine advantage over single-model approaches, where one misspecification biases the estimate.

The trade-off is transparency about weights. Because entropy balancing can assign large weights to comparison units that are scarce in a needed region of covariate space, extreme weights are possible, and the method assumes the balance constraints you chose are the ones that matter. It does not discover confounders; it balances the ones you name.

Why it matters for course evaluation in practice

Course-evaluation comparisons are observational almost by definition — you rarely randomise students to instructors or modalities — so confounding is the central threat, and this corpus documents its many faces: class size, discipline, difficulty, timing, self-selection. Entropy balancing helps in several concrete ways:

1. No data thrown away. In a faculty with 12 online sections and 40 in-person ones, propensity matching might discard most in-person sections to find matches. Entropy balancing keeps them all and reweights — a decisive advantage when your sample is already small.

2. Exact, checkable balance. After matching you must inspect balance tables and hope. After entropy balancing the specified moments are equal by construction; the balance table is a formality. That makes the adjustment easy to explain to a sceptical committee: "the comparison group has been reweighted to have the same average class size, same discipline mix, and same prior-GPA distribution as the group we are comparing it to."

3. Higher moments, not just means. You can balance variances as well as means — so the comparison group does not just have the same average class size but the same spread of class sizes. Distributional confounding, not only mean confounding, is addressed.

4. Fits the fairness use case. When the question is whether a modality or a redesign moved scores after fair adjustment — the same logic as difference-in-differences or propensity-score matching — entropy balancing gives a clean, reproducible weighting that a re-analysis will reproduce exactly, because there is no stochastic matching step.

Limitations and honest caveats

• It only balances measured confounders. Like every reweighting or matching method, entropy balancing does nothing about unobserved confounding. If motivated students self-select into the online section and motivation is unmeasured, balancing on class size and discipline will not fix the bias. This is exactly where a sensitivity analysis such as the E-value belongs — to ask how strong an unmeasured confounder would have to be to overturn the result.

• Extreme weights are a real risk. If the comparison group is thin where the treatment group is dense, exact balance forces very large weights onto a few units, inflating variance and making the estimate hinge on a handful of responses. Always inspect the weight distribution; a beautiful balance table built on three heavily weighted sections is fragile.

• You choose the constraints. The method balances the moments you specify. Forget a confounder and it stays unbalanced; the exactness is only as good as your covariate list. Entropy balancing is a disciplined executor of your causal assumptions, not a substitute for them.

• Double robustness is not triple robustness. Zhao and Percival's guarantee covers misspecification of the outcome or propensity model, not omission of a confounder or the wrong functional relationship between covariate and outcome. It is a real but bounded protection.

• Small samples still bite. Exact moment-balance in a comparison of two class sections of 20 is balancing on a knife-edge; the reweighted estimate can be unstable. Report uncertainty honestly — bootstrap the whole procedure — and resist over-interpreting a point estimate from thin data, the same discipline urged for small-cohort comparisons throughout this corpus.

How Koji incorporates this

Koji for Education is designed so that fair comparison is a first-class output, not an afterthought bolted on in a spreadsheet:

• The covariates that balancing needs are captured at source. Class size, delivery mode, discipline, cohort, and level are attached to every evaluation Koji collects. Entropy balancing is only as good as the confounder list you can supply; Koji's structured metadata is designed to make that list rich rather than forcing analysts to reconstruct it.

• Reweighted, not raw, comparisons. Where Koji compares groups — modalities, redesigns, cohorts — the reporting philosophy is to adjust for the measurable confounders and present an adjusted contrast with its uncertainty, rather than a raw mean difference that conflates teaching with class composition. Entropy balancing is one principled way to produce that adjustment reproducibly.

• Transparency about what was and was not adjusted. Because unmeasured confounding is the method's blind spot, Koji is designed to state which factors were balanced and to frame the adjusted contrast as "fair with respect to the measured factors" — designed to mitigate, not eliminate, confounding. Pairing the adjusted estimate with a sensitivity statement is the honest reporting the research demands.

• Beyond the number, to the why. Balancing tells you the adjusted score gap; it does not tell you why it exists. Koji's AI-moderated conversational interviews are built to probe the mechanism behind a difference, so an adjusted contrast can be read alongside the qualitative reasons students give. Koji's core research platform at koji.so applies the same engine to product and customer research, where comparing self-selected user groups raises identical confounding problems.

Related resources

• Propensity Score Matching for Course-Evaluation Confounds
• Selection Bias in Course Evaluations
• The E-Value for Unmeasured Confounding
• Difference-in-Differences for a Teaching Change
• Multilevel Models for Nested Course-Evaluation Data
• Funnel Plots for Fair Instructor Comparison

References

• Hainmueller, J. (2012). Entropy balancing for causal effects: A multivariate reweighting method to produce balanced samples in observational studies. Political Analysis, 20(1), 25-46. https://doi.org/10.1093/pan/mpr025
• Hainmueller, J., & Xu, Y. (2013). ebalance: A Stata package for entropy balancing. Journal of Statistical Software, 54(7), 1-18. https://doi.org/10.18637/jss.v054.i07
• Zhao, Q., & Percival, D. (2017). Entropy balancing is doubly robust. Journal of Causal Inference, 5(1), 20160010. https://doi.org/10.1515/jci-2016-0010
• Stuart, E. A. (2010). Matching methods for causal inference: A review and a look forward. Statistical Science, 25(1), 1-21. https://doi.org/10.1214/09-STS313
• Ho, D. E., Imai, K., King, G., & Stuart, E. A. (2007). Matching as nonparametric preprocessing for reducing model dependence in parametric causal inference. Political Analysis, 15(3), 199-236. https://doi.org/10.1093/pan/mpl013

Frequently asked questions

How is entropy balancing different from propensity-score matching?

Propensity-score matching estimates a model of the probability of being in the treatment group, then matches or weights cases on it and checks afterward whether covariates balanced — often re-specifying the model until they do, and discarding unmatched cases. Entropy balancing instead solves directly for weights that make the specified covariate moments exactly equal across groups in one step, keeping all cases and requiring no iterative balance search.

What does "doubly robust" mean here?

Zhao and Percival (2017) proved that the entropy-balancing estimate is consistent if either the outcome model or the implicit propensity model is correctly specified — you do not need both to be right. That is a stronger guarantee than single-model methods, where one misspecification biases the estimate. It does not protect against omitting a confounder entirely.

Does entropy balancing fix unmeasured confounding?

No. It balances only the covariates you supply. If an important confounder such as student motivation is unmeasured, balancing on class size and discipline will not remove the bias it causes. This is why an adjusted estimate should be paired with a sensitivity analysis such as the E-value, which asks how strong an unmeasured confounder would need to be to overturn the finding.

What are extreme weights and why should I worry about them?

To achieve exact balance when the comparison group is scarce in a region of covariate space, entropy balancing may assign very large weights to a few comparison units. The estimate then rests on those few responses, inflating its variance and fragility. You should always inspect the weight distribution; exact balance built on a handful of heavily weighted sections is not trustworthy.

Can I use it to compare just two class sections?

You can, but cautiously. Exact moment-balancing across two small sections is balancing on a knife-edge, and the reweighted estimate can be unstable. Bootstrap the whole procedure to get honest uncertainty, and avoid over-reading a point estimate drawn from thin data.

Where does Koji use this kind of adjustment?

Koji captures the confounders — class size, modality, discipline, cohort — that any balancing method needs, and its reporting is designed to present adjusted rather than raw group comparisons, with an explicit statement of what was balanced and the caveat that unmeasured factors may remain. The aim is a reproducible, fair contrast, not a raw mean difference that conflates teaching with class composition.