Random Forests and Variable Importance for Course-Evaluation Drivers

In brief

A random forest is a machine-learning method that averages hundreds of regression trees to predict an outcome — the overall course rating — from many teaching items at once, and then reports a variable-importance score for each item. Unlike a linear regression coefficient, it captures nonlinear thresholds and interactions without being told to look for them, which is why it can flag drivers that ordinary models miss. But the default (Gini/impurity) importance is systematically biased toward correlated predictors and variables with many categories, so a defensible course-evaluation analysis uses conditional permutation importance (Strobl et al. 2008) and treats every importance ranking as predictive association, never proof of cause.

What the research says

The method rests on three linked papers. Breiman, Friedman, Olshen and Stone (1984) introduced classification and regression trees (CART): recursively split the data on the item and threshold that most reduces prediction error, producing an interpretable but unstable tree. Breiman (2001) fixed the instability with the random forest — grow many trees, each on a bootstrap sample of respondents and a random subset of items at each split, then average their predictions. He proved the generalization error converges as trees are added and depends on individual-tree strength and the correlation between trees, and he proposed two importance measures: mean decrease in impurity (Gini) and permutation importance (shuffle one item across respondents, measure how much out-of-bag accuracy drops; a big drop means the item mattered).

Strobl, Boulesteix, Zeileis and Hothorn (2007, BMC Bioinformatics 8:25) showed the catch: standard variable-importance measures are biased. Impurity importance favours predictors with many possible split points (continuous or many-category items), and the unconditional permutation scheme inflates the apparent importance of items that are merely correlated with a truly influential item. In course evaluation, where "the lecturer was clear", "the lecturer was well-prepared" and "the lecturer explained difficult ideas" are all strongly intercorrelated, this bias is not hypothetical — it can reorder your driver list.

Strobl, Boulesteix, Kneib, Augustin and Zeileis (2008, BMC Bioinformatics 9:307) supplied the fix: conditional permutation importance, which permutes each item within strata defined by correlated items, so an item is credited only for the unique predictive signal it adds beyond its neighbours. This is the version to report when items overlap — which, in a well-designed evaluation instrument, they always do. The catR-adjacent ecosystem of R packages (randomForest, party/cforest, ranger) implements both schemes; the party implementation of conditional inference forests is the one Strobl et al. built for unbiased importance.

Why it matters for course evaluation in practice

Every quality office eventually asks the driver question: of the dozen things we measure, which actually move the overall rating? The default reflex is to run a linear regression of overall score on the item scores and read the standardized betas. That fails in two ways a forest handles. First, collinearity: when items are correlated, regression coefficients become unstable and can flip sign, so "preparation" may get a negative weight simply because "clarity" absorbed the shared variance. Second, nonlinearity and interaction: satisfaction with feedback may matter enormously below a threshold and not at all above it, or clarity may only drive the score when workload is high. A forest models these automatically and its partial-dependence plots show the shape, letting you see, for example, that assessment-fairness has a floor effect — ratings collapse below a point, then plateau.

Used honestly, this reframes an action plan. Instead of "raise every item", you learn which two or three behaviours carry the predictive weight and where the threshold sits, so a department can target the intervention that will actually shift the outcome. It also disciplines over-interpretation: if the forest explains little out-of-bag variance, that itself is a finding — the overall rating is not a tidy function of your items, and chasing item-level fixes is misguided.

A worked example makes the payoff concrete. Suppose a large first-year module collects ten teaching items and an overall rating from 180 students. A linear model reports clarity, preparation and organisation as three modest, unstable coefficients that shift whenever any one of them is dropped. A conditional random forest instead shows that assessment-fairness carries by far the most unique importance, that its partial-dependence curve is flat above the scale midpoint and falls steeply below it, and that clarity matters chiefly in the subset of students who also rated the workload as high. That single picture reframes the action plan: protect assessment-fairness above the threshold, and address clarity specifically where workload is heavy — a targeted response the averaged coefficients could never have justified.

Limitations and honest caveats

Four cautions keep this credible with a critical reader. Importance is not cause. A forest learns association in observational ratings; a high-importance item may be a proxy for an unmeasured confound (prior interest, grading leniency) rather than a lever. Everything in the platform is Simpson's paradox territory until a design supports causal claims. The bias is real and version-dependent — if you report default impurity importance you will over-credit your most granular, most correlated items; conditional permutation importance (Strobl et al. 2008) is the minimum defensible choice, and you must state which you used. Forests need data. With thirty responses and twelve items the trees overfit and importance rankings are unstable across bootstrap runs; report bootstrap confidence bands on importances and treat a small-class forest as exploratory. Interpretability is partial. A forest is not a set of rules; partial-dependence and SHAP-style plots approximate its behaviour but can mislead when items interact strongly. None of this makes the method unusable — it makes it a hypothesis generator that a multiple-comparisons mindset and a follow-up quasi-experiment should confirm.

How Koji incorporates this

Koji is designed to make driver analysis honest rather than to hand back a single tidy number. Its structured questions (open_ended, scale, single_choice, multiple_choice, ranking, yes_no) produce the mixed-type item matrix a forest needs, and its analysis layer is built to run conditional permutation importance, not just default impurity importance, so correlated clarity/preparation/organisation items are not double-counted. Importance rankings are reported with bootstrap stability bands and an out-of-bag fit statistic, so a thin or noisy dataset is flagged as exploratory instead of being dressed up as a definitive ranking. Because Koji captures open-text alongside the scales, a driver the forest flags can be read against what students actually wrote — the AI-moderated conversational interview probes why an item matters, turning a statistical association into an interpretable mechanism. Koji frames the output as "designed to surface candidate drivers for review", never as proof that raising an item will raise the score, and it points teams toward mid-cycle collection so a flagged driver can be tested with a change rather than assumed. Koji's core research platform at koji.so applies the same importance-analysis engine to product and customer research, where the "which feature drives satisfaction" question is identical in structure.

Frequently asked questions

Is a random forest better than linear regression for finding course-evaluation drivers?

Not universally — it is better when items are collinear or relationships are nonlinear/interacting, which is typical of evaluation data. Linear regression stays preferable when you need interpretable coefficients, a formal significance test, or you have too few responses for a forest to be stable. Run both and compare.

What is the difference between Gini importance and permutation importance?

Gini (mean decrease in impurity) importance sums how much each item reduces prediction error at splits; it is fast but biased toward continuous and many-category items. Permutation importance shuffles an item and measures the accuracy drop, which is fairer, and conditional permutation importance shuffles within correlated strata so an item is credited only for its unique signal.

Why does correlation among items distort variable importance?

Because the default permutation scheme lets a truly influential item lend its importance to any item correlated with it, and the tree-building step preferentially selects correlated predictors. Strobl et al. (2008) showed conditional permutation removes most of this artefact.

How many responses do I need before a forest is trustworthy?

There is no fixed cut-off, but with few dozen responses importance rankings vary wildly across bootstrap runs. Report bootstrap confidence intervals on importances; if they overlap heavily, treat the ranking as unreliable and collect more data.

Does high variable importance mean improving that item will raise the overall score?

No. Importance is predictive association in observational data. The item may be a proxy for a confound. To claim a lever, you need a design — a mid-term change, a comparison group, or a quasi-experiment — not a forest alone.

Can random forests handle open-text feedback?

Not directly; forests need structured features. You can first convert open-text into features (theme presence, sentiment) and feed those in, but interpret cautiously — the features inherit any coding error from the text pipeline.

Related Resources

References

Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1984). Classification and Regression Trees. Wadsworth. ISBN 978-0412048418.
Breiman, L. (2001). Random forests. Machine Learning, 45(1), 5–32. https://doi.org/10.1023/A:1010933404324
Strobl, C., Boulesteix, A.-L., Zeileis, A., & Hothorn, T. (2007). Bias in random forest variable importance measures: illustrations, sources and a solution. BMC Bioinformatics, 8, 25. https://doi.org/10.1186/1471-2105-8-25
Strobl, C., Boulesteix, A.-L., Kneib, T., Augustin, T., & Zeileis, A. (2008). Conditional variable importance for random forests. BMC Bioinformatics, 9, 307. https://doi.org/10.1186/1471-2105-9-307

Which Teaching Behaviours Drive the Overall Score — Nonlinearly? Random Forests and Variable Importance for Course Evaluation