Count Models for Course Evaluation: Why You Should Not Average the Number of Comments

In brief

When a course-evaluation variable is a count — the number of open-text comments a student left, how many times a theme appears, the number of complaints about a module — it is bounded at zero, skewed, and often piled up on zero. Averaging such counts or running ordinary regression on them produces biased standard errors and impossible predictions (like -0.4 comments). Poisson regression is the correct starting point; negative-binomial regression handles the overdispersion that is almost always present; and zero-inflated or hurdle models handle the excess zeros that come from students who would never comment at all. Choosing the right count model changes which differences look real.

What the research says

The statistical case is settled and well summarised for non-specialists. Coxe, West, and Aiken (2009), in a widely cited Journal of Personality Assessment tutorial, show that when an outcome is a count with a low mean (typically under about 10), ordinary least squares regression is biased and inefficient: it assumes a symmetric, constant-variance, unbounded outcome, whereas counts are asymmetric, have variance that grows with the mean, and cannot go below zero. Their recommendation is a graded family — start with Poisson, move to negative binomial when the data are overdispersed, and use zero-inflated variants when zeros are more common than a Poisson process would produce.

The Poisson model assumes the mean equals the variance (equidispersion). Real behavioural counts almost never satisfy this; they are overdispersed, with variance exceeding the mean because of unobserved heterogeneity between respondents. Ignoring overdispersion leaves the point estimates roughly right but shrinks the standard errors, so tests reject the null too often — you see "significant" differences that are noise. The negative-binomial model adds a dispersion parameter that absorbs this extra variance and restores honest standard errors.

Excess zeros are a separate problem. Diane Lambert (1992), in the foundational Technometrics paper on zero-inflated Poisson (ZIP) regression, modelled a population as a mixture: a subgroup that is structurally always zero (defects that cannot occur; students who will never write a comment) plus a Poisson-distributed subgroup whose count can be zero by chance. A ZIP or zero-inflated negative-binomial model fits two linked sub-models — a logistic model for "always-zero membership" and a count model for the rest — and thereby separates "no comment because disengaged" from "no comment this time." The hurdle model is the close cousin: a first stage for zero-vs-any and a truncated count model for the amount, given at least one. Atkins and Gallop (2007), writing for family-psychology researchers with the same infrequent-outcome problem course evaluators face, give an accessible tutorial and stress that the choice between zero-inflated and hurdle models should follow the substantive story about why the zeros arise.

Why it matters for course evaluation in practice

Course evaluation is full of counts that quality offices routinely mishandle by averaging.

Comment volume. "Course A averaged 0.8 comments per respondent, Course B 1.3" invites a t-test that is simply wrong for a variable that is zero for most students and occasionally three or four. A negative-binomial model gives a defensible comparison and a rate ratio ("students left 60% more comments in B") that is both correct and easy to communicate.

Theme incidence. If automatic coding tags how many distinct concerns each student raised, that is a count. Modelling it as a count reveals whether a redesign genuinely reduced the number of problems students name, rather than just nudging a mean.

Complaints and rare events. The number of students flagging a specific serious issue is a low-count, zero-heavy variable — exactly the regime where OLS invents negative predictions and Poisson underestimates the spread. A zero-inflated model can even tell you whether an intervention worked by reducing the always-zero barrier (more students willing to speak up) versus reducing the rate (fewer issues among those who do), which are different quality signals.

Counts of scale endorsements. How many of ten teaching-behaviour items a student marked "always" is a bounded count that behaves far better under a count or binomial model than under an average.

Getting the model right is not pedantry: overdispersion inflates false positives, so a naive analysis will keep flagging noise as a real drop in engagement.

Limitations and honest caveats

Count models are the right tool, but they carry their own traps a careful reviewer will name.

First, model selection can be driven by fit-fishing. With Poisson, negative-binomial, ZIP, hurdle, and zero-inflated negative-binomial all on the table, it is tempting to pick whichever gives the result you want. Discipline this with the substantive question (is there a genuine structural-zero group?), a likelihood-ratio or Vuong-style comparison used cautiously, and information criteria — decided before looking at the coefficient of interest.

Second, zero-inflation and overdispersion are confusable. Excess zeros can themselves create apparent overdispersion, so a negative-binomial model sometimes fits excess-zero data adequately without a separate inflation component. Do not add a zero-inflation stage unless the data and the theory both call for it; the simpler model is often better.

Third, counts depend on exposure. A student who answered more open-text prompts has more opportunity to comment. Count models handle this with an offset (the log of exposure), but forgetting the offset silently confounds volume with opportunity.

Fourth, rare events mean low power and unstable estimates. With few nonzero observations, zero-inflated models can fail to converge or produce wild parameters. Sometimes the honest answer is a simple reported proportion with a confidence interval, not a two-part model the data cannot support.

Finally, a count model tells you about quantity, not quality or valence. More comments is not better or worse teaching; pair count models with sentiment and thematic analysis to interpret what the counts mean.

How Koji incorporates this

Koji is built so the counts are meaningful and well-defined before anyone models them, which is where most count analyses go wrong.

• Clean exposure and structured counts. Because Koji records exactly which prompts each student saw and answered, the exposure needed for a correct offset is available, and counts of endorsements across scale, multiple_choice, and yes_no items are captured in a tidy schema rather than reconstructed from free text.
• Theme counts from conversational depth. Koji's AI-moderated interview probes beyond a single Likert number, and its automatic thematic analysis turns each transcript into a defensible count of distinct concerns raised — the exact input a negative-binomial or hurdle model needs to test whether a redesign reduced the number of problems students name.
• Separating the two kinds of silence. Koji distinguishes a student who declined to elaborate from one who engaged but had nothing to add, which maps directly onto the structural-zero-versus-sampling-zero distinction at the heart of zero-inflated and hurdle models — so a quality office can tell "we made students more willing to speak" apart from "we reduced the underlying rate of complaints."
• Honest small-count reporting. When nonzero counts are too few to support a two-part model, Koji reports a proportion with an interval and flags the instability rather than over-fitting, consistent with its small-sample-aware philosophy.

Koji's core research platform at koji.so applies the same engine to product and customer research, where counts of feature requests or support contacts raise identical modelling questions — the education product simply points it at teaching and courses.

Frequently asked questions

Why not just average the number of comments per student?

Because a count is bounded at zero, skewed, and has variance that grows with the mean, so a mean and a t-test give biased standard errors and can imply impossible negative values. A Poisson or negative-binomial model respects the shape of the data and reports an interpretable rate ratio instead of a fragile mean difference.

When do I need a negative-binomial model instead of Poisson?

When the data are overdispersed — variance greater than the mean, which is almost always true for behavioural counts. Poisson then understates the standard errors and produces too many false positives. A negative-binomial model adds a dispersion parameter that fixes this; check the dispersion statistic or compare the two models formally.

What is the difference between zero-inflated and hurdle models?

Both handle excess zeros with two parts, but a zero-inflated model treats zeros as a mixture of structural "never" zeros and ordinary count zeros, while a hurdle model treats all zeros as one gate that must be crossed before a truncated count applies. Choose based on whether a genuine always-zero group exists in your setting.

How do I know if I have excess zeros at all?

Compare the observed proportion of zeros with the proportion a fitted Poisson would predict; a large gap signals zero-inflation. But note that overdispersion alone can create extra zeros, so a negative-binomial model sometimes fits without a separate inflation stage. Let both theory and diagnostics decide.

What is an offset and why does it matter?

An offset is the log of each respondent's exposure — for example, how many prompts they could have commented on — included so the model estimates a rate per opportunity rather than a raw count. Omitting it confounds "commented more" with "had more chances to comment," a common and serious error in comment-volume analysis.

Do count models tell me whether feedback is positive or negative?

No. Count models describe how many, not how good or bad. A rise in comment volume could reflect enthusiasm or frustration. Always pair a count model with sentiment and thematic analysis so the quantity is interpreted in light of what students actually said.

Related resources

• Ordinal Regression: Why Cumulative-Link Models Beat Averaging Likert Scores
• The Average Hides the Tails: Quantile Regression for Course-Evaluation Data
• Robust Estimators for Course Evaluation
• Ceiling Effects, Skew, and What They Do to Course-Evaluation Statistics
• How Many Open-Text Comments Are Enough? Thematic Saturation
• Top-Box vs Mean: Reporting Scores Without Throwing Away Information

References

• Lambert, D. (1992). Zero-inflated Poisson regression, with an application to defects in manufacturing. Technometrics, 34(1), 1-14. https://doi.org/10.2307/1269547
• Coxe, S., West, S. G., & Aiken, L. S. (2009). The analysis of count data: A gentle introduction to Poisson regression and its alternatives. Journal of Personality Assessment, 91(2), 121-136. https://doi.org/10.1080/00223890802634175
• Atkins, D. C., & Gallop, R. J. (2007). Rethinking how family researchers model infrequent outcomes: A tutorial on count regression and zero-inflated models. Journal of Family Psychology, 21(4), 726-735. https://doi.org/10.1037/0893-3200.21.4.726
• Cameron, A. C., & Trivedi, P. K. (2013). Regression Analysis of Count Data (2nd ed.). Cambridge University Press. https://doi.org/10.1017/CBO9781139013567