Some of Your Course-Evaluation Responses Are Careless: Detecting Insufficient-Effort Responding

In brief

In almost every course-evaluation dataset, a minority of responses are not genuine judgements of teaching — they are produced by students clicking through without reading the items. This is called insufficient-effort or careless responding (IER/C). The most-cited methodological study, Meade and Craig (2012), found that roughly 10–12% of respondents on a long survey showed careless patterns, and demonstrated that several non-intrusive indices — long-string, response time, psychometric synonyms/antonyms, and multivariate (Mahalanobis) outlier distance — can flag them. For course evaluation, screening for careless responses before you compute and report means is a cheap, transparent way to protect data quality, and it matters more as response shifts to low-stakes, mobile, end-of-term completion.

What the research says

Careless responding is the survey-methods term for answers given without adequate attention to item content. Andrew Meade and S. Bartholomew Craig's 2012 paper in Psychological Methods, "Identifying Careless Responses in Survey Data," is the field's reference point. Across two studies of undergraduates completing a lengthy questionnaire for course credit, they evaluated a battery of detection methods: special instructed-response items (e.g., "select strongly agree for this item"), consistency indices built from ordinary items (psychometric synonyms and antonyms, even–odd consistency), multivariate outlier analysis via Mahalanobis distance, response time, and self-reported diligence. Two findings travel directly to course evaluation. First, careless responding is not rare: about 10–12% of respondents were flagged. Second, there are two distinct patterns — broadly random responding and non-random (e.g., repetitive "straight-line") responding — and no single index catches both, so a small panel of indices outperforms any one of them.

The corroborating literature is consistent. Huang and colleagues (2012), in the Journal of Business and Psychology, compared response time, long-string, psychometric antonyms, and individual reliability coefficients, and showed that a simple warning to respondents (a deterrent) plus post-hoc screening improved data quality on low-stakes surveys — exactly the conditions of an end-of-term course evaluation. Curran (2016), in the Journal of Experimental Social Psychology, provides a practical taxonomy of indices and stresses that they should be chosen and (where possible) pre-registered before data collection, not fished for afterwards. The recent Annual Review of Psychology synthesis by Ward and Meade (2023) frames the problem in terms of prevention, identification, and best practice, recommending a combination of design choices (attention checks, reasonable length) and multi-index screening rather than reliance on a single magic number.

The key technical ideas, in plain terms:

• Long-string index: the longest run of identical consecutive answers. A student who selects "4" for twenty items in a row is either genuinely uniform or not reading; the index flags the latter for inspection.
• Response time / page time: completions far faster than is plausible for reading the items (often operationalised as below ~2 seconds per item) are suspect.
• Psychometric synonyms/antonyms and even–odd consistency: if a respondent answers two items that should correlate (or anti-correlate) in contradictory ways, internal consistency drops — a signal of inattention.
• Multivariate outlier (Mahalanobis) distance: flags response vectors that sit far from the multivariate centroid, catching patterns no univariate check would.
• Instructed-response / attention-check items: a directly worded item with a known correct answer.

Why it matters for course evaluation in practice

Course evaluation is the textbook setting for careless responding: the survey is low-stakes for the student, often completed on a phone in the last week of term, frequently after several other surveys, and sometimes under mild institutional pressure to "just submit something." Three practical consequences follow.

1. Careless responses distort the mean and the variance. Random responding pulls means toward the scale midpoint and inflates variance; straight-lining at the top inflates means and shrinks variance (a contributor to the ceiling effects many institutions see). Either way, a handful of inattentive vectors can move a small-class average enough to change how a committee reads it.

2. It corrupts subscale structure and reliability. If your instrument reports separate dimensions (clarity, organisation, assessment, feedback), careless rows weaken the very correlations that make those subscales meaningful, depressing reliability statistics and muddying any factor structure.

3. It contaminates open-text analysis. The same disengaged respondent who straight-lines the Likert grid often leaves an empty or junk free-text box — or, increasingly, a generic AI-pasted paragraph. Screening at the response level helps keep both the quantitative and qualitative layers clean.

The correct response is not to delete aggressively — over-zealous removal is its own bias — but to screen transparently: compute a small set of indices, set defensible thresholds in advance, flag rather than silently drop, and report how many responses were excluded and why. This is the same logic that quality-assurance officers already apply to small-sample reliability (see How Many Responses Do You Need for a Reliable Course Evaluation?).

Limitations and honest caveats

A critical reader should hold several objections in view. First, indices have false positives. A student in a genuinely excellent (or genuinely poor) course may legitimately rate every item at the ceiling (or floor); long-string and low-variance flags will catch them too. This is why flags should trigger inspection or sensitivity analysis, not automatic deletion. Second, thresholds are not universal. The "2 seconds per item" or "long-string > 10" rules of thumb depend on instrument length, item complexity, and reading load; they must be calibrated to your own form, ideally on historical data. Third, removing cases changes your sample and can introduce a different non-response/selection problem if careless responders differ systematically from attentive ones (a concern that connects to wave analysis of non-response bias). Fourth, generalisability: much of the foundational work used long research questionnaires completed for course credit, which are longer than a typical 8–12 item course evaluation; with shorter forms, long-string and consistency indices have less signal to work with, so design-side prevention (attention to length, see questionnaire length and data quality) becomes relatively more important. Finally, attention-check items can backfire: heavy-handed "gotcha" items can annoy conscientious students and are ethically awkward in an evaluation students are required to take. Subtle, design-based prevention is usually preferable to interrogation.

How Koji incorporates this

Koji is designed to reduce the conditions that produce careless responding in the first place, and to surface — rather than silently bury — the responses that remain low-effort.

• Conversational, AI-moderated interviews instead of a long Likert grid. The single biggest driver of straight-lining is a monotonous matrix of agree/disagree rows. Koji replaces much of that grid with a short, adaptive conversation that asks structured questions (open_ended, scale, single_choice, multiple_choice, ranking, yes_no) one at a time and follows up on what the student actually says. A respondent cannot "straight-line" a conversation that probes their previous answer, which structurally lowers the IER rate the literature documents.
• Quality scoring at the response level. Koji computes engagement and quality signals on each completed interview — response time, substantive length of open-text, and internal consistency between a student's narrative and their scale answers — and flags thin or contradictory submissions for review rather than dropping them invisibly. This operationalises the Meade and Craig recommendation to use multiple indices, not one.
• Probing beyond the number. Because the AI moderator can ask "you rated the assessment a 2 — what specifically made it hard?", a genuinely low score arrives with corroborating text, while a careless low score tends to produce an empty or evasive follow-up. That concordance check (see do written comments match the ratings?) is itself a careless-response signal.
• Bias-aware, transparent reporting. When Koji reports a course mean, the underlying flags travel with it, so an evaluation committee sees how many responses were low-engagement before reading the average — consistent with the responsible-reporting norms in Interpreting and Reporting Student Ratings Responsibly.

Koji frames these as mitigations, not cures: no platform eliminates careless responding, and Koji's quality flags are decision support for human reviewers, not an automatic deletion rule. The same AI-moderated interview engine powers Koji's core research platform at koji.so for product and customer research, where insufficient-effort responding is an equally well-known threat to survey validity.

A practical screening checklist

For a quality-assurance office adopting screening for the first time, a proportionate workflow looks like this. Decide your indices and thresholds before the collection cycle and write them into your reporting policy, so the rules cannot be reverse-engineered to produce a desired result. Compute a long-string index and a page-level response-time index for every submission, and add a consistency measure (psychometric synonyms or even–odd) whenever the instrument carries enough items to support one. Flag — do not delete — any response that trips two or more indices, and route flagged cases to a brief human check rather than an automatic deletion rule. Report the number of flagged and excluded responses alongside each course mean, together with the rationale, so the screening is fully auditable. Re-calibrate thresholds annually against your own historical distributions, because rules of thumb borrowed from long research questionnaires rarely fit a short evaluation form. Treated this way, screening becomes a transparent, defensible safeguard rather than a hidden act of data manipulation, and it gives an evaluation committee a documented basis for trusting the numbers it reads.

Related resources

• Why Students Click Straight Down the Middle: Satisficing in Course Evaluations
• How Long Should a Course Evaluation Be? Questionnaire Length, Breakoff, and Answer Quality
• Do Students Completing Evaluations on Their Phones Give Worse Data? The Device-Effects Evidence
• How Many Responses Do You Need for a Reliable Course Evaluation?
• Are Your Respondents Representative? Using Early-vs-Late Wave Analysis to Estimate Nonresponse Bias
• How Reliable Is Your Coding of Open-Text Feedback? Inter-Rater Reliability and Thematic Analysis

References

• Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17(3), 437–455. https://doi.org/10.1037/a0028085
• Huang, J. L., Curran, P. G., Keeney, J., Poposki, E. M., & DeShon, R. P. (2012). Detecting and deterring insufficient effort responding to surveys. Journal of Business and Psychology, 27(1), 99–114. https://doi.org/10.1007/s10869-011-9231-8
• Curran, P. G. (2016). Methods for the detection of carelessly invalid responses in survey data. Journal of Experimental Social Psychology, 66, 4–19. https://doi.org/10.1016/j.jesp.2015.07.006
• Ward, M. K., & Meade, A. W. (2023). Dealing with careless responding in survey data: Prevention, identification, and recommended best practices. Annual Review of Psychology, 74, 577–596. https://doi.org/10.1146/annurev-psych-040422-045007