Deleting Incomplete Surveys Throws Away Good Data: Multiple Imputation for Missing Course-Evaluation Responses

Answer first. When students complete a course evaluation but leave some items blank, the default of deleting every partially-complete response ("listwise deletion") both shrinks your sample and can bias the result. Multiple imputation (MI) is the well-established alternative: it fills each blank with several plausible values drawn from a model, analyses each completed dataset, and pools the results so that the extra uncertainty from having imputed is carried through honestly. Under the missing-at-random assumption, MI is more efficient and less biased than deletion — and, unlike single-value fill-ins, it does not pretend the guesses were real data.

What the research says

There are two distinct missing-data problems in course evaluation. Unit nonresponse is the student who never opens the survey. Item nonresponse is the student who answers most questions but skips a few — the workload item, a sensitive question about the instructor, or the open-text box. This article is about the second, and multiple imputation is the standard modern remedy.

The method was introduced by Rubin (1987) in Multiple Imputation for Nonresponse in Surveys. The core idea is deceptively simple. Rather than replace a missing value with a single best guess — which understates uncertainty because it treats an estimate as if it were an observation — MI replaces each missing value with m different plausible values (commonly m = 5 to 50), producing m complete datasets. You run your ordinary analysis on each, then combine the m sets of results using Rubin's rules: the pooled estimate is the average of the m estimates, and the pooled variance adds the within-imputation variance to the between-imputation variance. That second term is the whole point — it inflates standard errors to reflect the fact that the imputed values were uncertain.

Schafer and Graham (2002), in their landmark Psychological Methods review "Missing data: our view of the state of the art," established MI and maximum-likelihood methods as the defensible defaults for social-science data and made the case that the older reflexes — listwise deletion, pairwise deletion, and mean substitution — are usually inferior. Mean substitution in particular is shown to distort variances and correlations; it is the one method almost everyone should stop using. Enders (2010), in Applied Missing Data Analysis, and van Buuren's work on chained-equations imputation (MICE) turned MI from theory into routine practice, providing the algorithms that let you impute a mix of Likert, categorical, and continuous evaluation items conditionally on one another.

Within higher-education research specifically, Manly and Wells (2015) surveyed how missing data is actually handled in the field's journals and documented widespread under-reporting and over-reliance on deletion, arguing that MI should be reported transparently — including the imputation model, the number of imputations, and the missing-data assumption. Their paper is a useful corrective for QA offices that treat missing evaluation items as a nuisance to be dropped rather than a modelling decision to be justified.

Why it matters for course evaluation in practice

Listwise deletion is not the "safe" default. It feels conservative — "we only used complete responses" — but it has two costs. First, statistical power: dropping every response with any blank can discard a large fraction of a small class's already-thin data. Second, bias: deletion is only safe if the discarded responses are missing completely at random (MCAR), which is a strong and often false assumption. If students who felt negatively about the instructor are the ones who skipped the "overall satisfaction" item, deleting them biases the mean upward.

Item non-response is informative and patterned. Blanks cluster on sensitive items, on the last items of a long survey (fatigue), and on open-text boxes. Because the pattern is not random, the choice of how to handle it is a substantive analytic decision, not clerical hygiene.

MI keeps the uncertainty visible. The alternative "single imputation" — filling a blank with the item mean or a regression prediction — produces a tidy complete dataset but lies about precision, because it treats a guess as a fact and shrinks standard errors. MI's between-imputation variance is exactly the mechanism that prevents this false confidence. For a programme director deciding whether an instructor's score is meaningfully below benchmark, honest standard errors are the difference between a defensible flag and a spurious one.

The practical recipe: impute the missing items m times from a model that includes the other evaluation items and any useful auxiliary variables (grade, year, discipline), analyse each of the m datasets with your ordinary method, and pool with Rubin's rules. Report the missing-data rate, the assumption you relied on, and the number of imputations — the transparency Manly and Wells (2015) argue is routinely absent.

Limitations and honest caveats

• MI assumes MAR, not MNAR. Standard multiple imputation is valid under missing-at-random: blanks depend only on observed data. If students skip the satisfaction item because of their (unobserved) satisfaction, that is missing-not-at-random, and ordinary MI does not fix it — it requires explicit MNAR modelling and untestable assumptions. MI is a remedy for item nonresponse under MAR, not a universal solvent. (See the companion note on nonresponse mechanisms.)
• Garbage in, garbage out on the imputation model. MI is only as good as the model that generates the fills. Omitting variables that predict both the missingness and the outcome reintroduces bias; a badly specified imputation model can be worse than deletion. Auxiliary variables matter.
• Small samples strain the method. In a seminar of 12, an imputation model has little data to learn from, and the between-imputation variance can be large and unstable. MI does not manufacture information that was never collected; it propagates uncertainty, and in tiny classes that uncertainty may be so large that no confident conclusion is warranted — which is itself the correct answer.
• It is not a licence to over-survey. The best missing-data strategy is to prevent blanks in the first place through shorter, well-designed instruments. MI repairs damage; it should not excuse a bloated questionnaire that manufactures fatigue-driven missingness.
• Reproducibility requires seeds and documentation. Because imputation is stochastic, results shift slightly between runs unless the random seed and imputation specification are recorded. Undocumented MI is not reproducible MI.

How Koji incorporates this

Koji's stance is prevent first, model honestly second — the order the missing-data literature recommends.

• Prevention through design. Because item nonresponse is driven heavily by length and fatigue, Koji's short, AI-moderated conversational format is engineered to reduce the blanks that would otherwise require imputation. A conversation that adapts and does not force every student through an identical long grid produces fewer skipped items to begin with.
• Honest uncertainty in reporting. Where blanks remain, Koji is designed to represent scores with uncertainty intervals rather than as bare point estimates, aligning with the MI principle that imputed or incomplete data must not be reported with false precision. A programme director sees the width of the interval, not just the midpoint.
• Auxiliary-rich analysis. Koji captures structured metadata (question type, course context, response completeness) alongside each answer, which is exactly the auxiliary information a defensible imputation or weighting model needs to keep the MAR assumption plausible.
• Open-text recovery instead of blanks. The single most-skipped item on a traditional evaluation is the open-text box. Koji's AI-moderated interview probes conversationally for that qualitative content, converting what would be a missing free-text field into captured, thematically-analysed feedback — reducing the missingness rather than imputing around it. This is framed as designed to reduce item nonresponse, not to eliminate it.

Teams running product or customer research face the same partially-complete-survey problem; Koji's core platform at koji.so applies the identical adaptive-interview approach to keep response records complete in non-teaching studies.

Related resources

• Non-Response Weighting and Post-Stratification
• Planned Missingness and Split-Questionnaire Designs
• Careless and Insufficient-Effort Responding
• Multilevel Models for Nested Evaluation Data
• Bootstrap Confidence Intervals for Small Classes
• The Total Survey Error Framework

Frequently asked questions

What is multiple imputation, in one sentence?

Multiple imputation replaces each missing value with several plausible values drawn from a model, analyses each resulting complete dataset separately, and pools the results with Rubin's rules so that the uncertainty of having imputed is carried into the final standard errors rather than hidden.

Why not just delete responses that have blanks?

Listwise deletion discards usable data — halving the power of an already-small class — and is only unbiased if the blanks are missing completely at random, which is rarely true. If students who disliked the course skipped the satisfaction item, deletion biases the mean. Multiple imputation retains the partial responses and, under the weaker missing-at-random assumption, is both more efficient and less biased.

Why is filling blanks with the item mean a bad idea?

Mean substitution treats a guess as a real observation. It artificially shrinks the variance and distorts correlations between items, producing standard errors that are too small and confidence that is unwarranted. Schafer and Graham (2002) single it out as a method to avoid; multiple imputation exists precisely to preserve the lost uncertainty.

Does multiple imputation fix a low overall response rate?

No. MI addresses item nonresponse — blanks within otherwise-completed surveys — under the missing-at-random assumption. It does not recover the opinions of students who never responded at all (unit nonresponse), and it does not fix missing-not-at-random mechanisms where the blank depends on the unobserved answer itself.

How many imputations should I create?

Rubin's early guidance suggested as few as 3–5; modern practice, reflecting the availability of computing power and the desire for stable standard errors, commonly uses 20–50, especially when the fraction of missing information is high. The key is to report the number used alongside the imputation model and the missing-data assumption.

References

• Rubin, D. B. (1987). Multiple Imputation for Nonresponse in Surveys. Wiley. https://doi.org/10.1002/9780470316696
• Schafer, J. L., & Graham, J. W. (2002). Missing data: Our view of the state of the art. Psychological Methods, 7(2), 147–177. https://doi.org/10.1037/1082-989X.7.2.147
• Enders, C. K. (2010). Applied Missing Data Analysis. Guilford Press.
• van Buuren, S. (2018). Flexible Imputation of Missing Data (2nd ed.). CRC Press. https://doi.org/10.1201/9780429492259
• Manly, C. A., & Wells, R. S. (2015). Reporting the use of multiple imputation for missing data in higher education research. Research in Higher Education, 56(4), 397–409. https://doi.org/10.1007/s11162-014-9344-9