Which Variables Should You Actually Control For? Causal Diagrams and the Backdoor Criterion for Course Evaluation

In brief

Adding another variable to a course-evaluation comparison is not automatically "more rigorous". Whether a control removes bias or creates it depends entirely on the variable's role in the causal structure — confounders should be adjusted for, but mediators and colliders should not, and adjusting for the wrong one moves your estimate away from the truth. A causal diagram (a directed acyclic graph) plus Pearl's backdoor criterion gives you a defensible, before-you-run-the-analysis rule for choosing exactly which variables belong in the model.

What the research says

The formal machinery comes from Judea Pearl's Causal diagrams for empirical research (Pearl, 1995, Biometrika 82(4):669-688), which showed that a directed acyclic graph (DAG) encoding your subject-matter assumptions can be queried mathematically to decide whether a causal effect is identifiable from observational data — and, if so, which set of variables to condition on. Greenland, Pearl and Robins (1999, Epidemiology 10(1):37-48) translated this into applied practice, giving researchers a set of graphical rules for reading confounding, selection bias and mediation directly off a diagram.

The single most important rule is the backdoor criterion. An estimate of the effect of X (say, a teaching change) on Y (the rating) is unbiased if the variables you condition on (a) block every "backdoor" path — every non-causal path from X to Y that starts with an arrow into X — and (b) do not include any descendant of X. In plain terms: adjust for common causes of both the intervention and the outcome, and never adjust for anything that sits on the causal pathway after the intervention.

Cinelli, Forney and Pearl (2024, Sociological Methods & Research 53(3):1071-1104, "A Crash Course in Good and Bad Controls") catalogues the failure modes with a series of minimal diagrams. Their taxonomy is the practical payload: a "good control" closes a backdoor path and reduces bias; a "bad control" is a mediator (adjusting for it removes part of the very effect you want), a collider (adjusting for it opens a spurious path — the M-bias structure), or a variable that induces over-control or selection bias. Critically, they demonstrate cases where adding a plausible-sounding covariate increases bias — the opposite of the "control for everything" instinct.

Rohrer (2018, Advances in Methods and Practices in Psychological Science 1(1):27-42) makes the same point for a social-science audience: colliders and mediators "should not be controlled for because that can actually move the estimate of an association away from the value of the causal effect of interest." She stresses that statistical control is not a neutral act of tidying — it is a causal assumption in disguise, and the DAG is where that assumption becomes explicit and criticisable.

Why it matters for course evaluation in practice

Almost every non-trivial claim a quality office makes is causal in disguise: this redesign raised satisfaction, small classes score higher because they are small, the new instructor is genuinely weaker. Answering any of these from observational evaluation data requires choosing what to adjust for, and the field's default — throw class size, discipline, level and cohort into a regression and report the "adjusted" score — is exactly the move Cinelli and colleagues warn about.

Two concrete traps recur. First, the mediator trap: if you believe a teaching change works by increasing student engagement, then controlling for an engagement measure will absorb the effect and make a real improvement look like nothing. Second, the collider trap: conditioning on who chose to respond, or on which students remained enrolled, can manufacture correlations between instructor quality and student characteristics that do not exist in the population — the structure behind our companion article on collider bias.

Drawing the DAG first changes the workflow. You commit, in advance, to a diagram of who-affects-what — grading leniency, prior interest, discipline, timetable slot, response propensity — and then read off the minimal sufficient adjustment set. The set is often smaller than the kitchen-sink model, and occasionally it tells you the effect is not identifiable at all from the data you have, which is a more honest answer than a confident but biased coefficient. This is the same discipline that underpins propensity score matching and entropy balancing: those methods estimate an effect given an adjustment set; the DAG is how you justify the set in the first place.

Limitations and honest caveats

A DAG is only as good as the assumptions drawn into it. The graph does not discover causal structure from data; it encodes your beliefs, and a missing arrow is a strong claim that two variables are conditionally independent. Two analysts can draw different defensible diagrams and reach different adjustment sets — the DAG makes the disagreement explicit but does not resolve it.

The backdoor criterion also assumes the confounders you name are actually measured. If a common cause (say, unrecorded student motivation) is absent from your dataset, no adjustment set can close its path, and the honest conclusion is residual confounding — which is precisely where sensitivity analysis such as the E-value and the negative-control diagnostics take over. Finally, DAGs are non-parametric about structure but silent about functional form; a correctly specified adjustment set can still be undermined by a badly specified model. The diagram is a necessary discipline, not a sufficient one.

How Koji incorporates this

Koji is designed to make the "what should we compare, and what should we hold constant" decision explicit rather than accidental. When you frame an evaluation as a comparison — a redesigned course against its prior cohorts, one delivery mode against another — Koji's structured question types (single_choice, scale, yes_no) let you capture the covariates a defensible DAG actually needs (discipline, level, prior interest, workload perception) as first-class fields rather than free-text afterthoughts, so an adjustment set is available at analysis time.

Just as importantly, Koji's AI-moderated conversational interview is built to probe mechanism — the why behind a rating — which is what tells you whether a candidate covariate is a confounder or a mediator. If students consistently explain that a change helped because it improved their engagement, that is evidence the engagement variable sits on the causal path and should be left out of an adjustment for the change's total effect. Koji's reporting frames adjusted comparisons as assumption-dependent, and its bias-aware reporting is designed to flag when a comparison rests on covariates that may be post-treatment. Koji does not draw your DAG for you — causal assumptions are a human, subject-matter responsibility — but it is built to surface the information a credible diagram requires and to avoid presenting an "adjusted" number as if adjustment were self-evidently neutral. Teams running broader product or customer research on Koji's core platform at koji.so use the same discipline when they ask whether a controlled variable is a confounder or a consequence.

Frequently asked questions

What is the backdoor criterion in one sentence?

Adjust for a set of variables that blocks every non-causal path from the intervention to the outcome (every path with an arrow pointing into the intervention) while including no variable that is itself a consequence of the intervention.

Isn't controlling for more variables always safer?

No. Controlling for a mediator removes part of the real effect, and controlling for a collider opens a spurious association — both move your estimate away from the truth. "Adjust for everything" is a recipe for bad controls, as Cinelli, Forney and Pearl (2024) demonstrate.

What is the difference between a confounder, a mediator and a collider?

A confounder is a common cause of both the intervention and the outcome (adjust for it). A mediator lies on the causal path from intervention to outcome (do not adjust if you want the total effect). A collider is a common effect of two variables (adjusting for it induces bias).

Do I need special software to draw a DAG?

No — the value is in the reasoning, and a diagram on paper is enough to derive an adjustment set for a simple problem. Free tools such as DAGitty automate the backdoor search for larger graphs, but the discipline of committing to the arrows first is what matters.

What if a confounder is unmeasured?

Then no adjustment set can remove its bias, and you should say so. Pair the DAG with sensitivity analysis (for example the E-value) or a negative-control diagnostic to bound how much unmeasured confounding could distort your conclusion.

How is this different from propensity scores or matching?

Those are estimation techniques that balance a given set of covariates. The DAG and backdoor criterion tell you which covariates that set should contain. You need both: the diagram to choose the variables, the matching or weighting to condition on them.

References

• Pearl, J. (1995). Causal diagrams for empirical research. Biometrika, 82(4), 669-688. https://doi.org/10.1093/biomet/82.4.669
• Greenland, S., Pearl, J., & Robins, J. M. (1999). Causal diagrams for epidemiologic research. Epidemiology, 10(1), 37-48. https://doi.org/10.1097/00001648-199901000-00008
• Cinelli, C., Forney, A., & Pearl, J. (2024). A crash course in good and bad controls. Sociological Methods & Research, 53(3), 1071-1104. https://doi.org/10.1177/00491241221099552
• Rohrer, J. M. (2018). Thinking clearly about correlations and causation: Graphical causal models for observational data. Advances in Methods and Practices in Psychological Science, 1(1), 27-42. https://doi.org/10.1177/2515245917745629

Related resources

• Collider Bias: How Selecting Who Gets Evaluated Invents Correlations That Are Not There
• Propensity Score Matching for Course-Evaluation Confounds
• How Strong Would the Hidden Confounder Have to Be? The E-Value for Adjusted Course-Evaluation Comparisons
• Selection Bias in Course Evaluations: What Goos and Salomons Found
• Why Was the Course Rated Well, Not Just Whether? Mediation Analysis and the Mechanism Behind an Evaluation Score
• Exactly Balanced Comparison Groups Without the Guesswork: Entropy Balancing for Course Evaluation