Do Students Completing Evaluations on Their Phones Give Worse Data? The Device-Effects Evidence

In brief: The device a student uses does affect course-evaluation data — but mostly at the margins that good design can fix, not at the core. Champagne (2013), analysing more than 400,000 course-evaluation responses, found significant device differences in response rate and in the number of comments typed, but no significant difference in the actual instructor and course ratings. The wider survey-methodology evidence agrees that the main mobile penalty is shorter open-text answers and occasional breakoff, and that mobile-optimised, easy-to-tap question formats largely close even those gaps (Antoun, Couper & Conrad, 2017). The risk is not that phones produce wrong numbers; it is that a non-mobile-friendly form quietly shortens the qualitative feedback you most want.

The question behind a now-universal behaviour

A decade ago, most course evaluations were completed on a laptop or in a computer lab. Today a large share of students answer on a phone, often in the minutes after class or on a commute. That shift raises a fair institutional worry: if the survey was designed for a desktop screen, are smartphone responses systematically worse — lower-quality ratings, abandoned surveys, one-word comments? The answer matters because evaluation scores feed module review and accreditation evidence, and because the open-text comments are where teachers find the most actionable feedback. The good news is that this question has been studied directly, including in the course-evaluation context itself.

What the research says

The most on-point study is Champagne (2013), "Student Use of Mobile Devices in Course Evaluation," in Educational Research and Evaluation. This is not a marketing panel — it is real course-evaluation data: over 400,000 responses across 95 campuses over 20 months, comparing submissions from phones, tablets, and computers. Two results are central. First, device type did significantly affect response rate and the number of comments students typed — phone users behaved differently in how much they wrote and how readily they participated. Second, and reassuringly, device type did not significantly affect the mean ratings of the instructor and the course. The numbers institutions report up the chain were stable across devices; what shifted was the richness of the qualitative feedback and the propensity to respond.

The general survey-methodology literature reaches a compatible verdict. Mavletova (2013), "Data Quality in PC and Mobile Web Surveys" in Social Science Computer Review, ran a two-mode experiment and found that mobile completion was associated with lower completion rates and shorter answers to open-ended questions, with some response-order effects — i.e., the costs concentrate in participation and open-text depth, not in the substantive distribution of closed answers.

Crucially, the more recent and better-powered work suggests these penalties are largely a design problem, not an inherent property of phones. Antoun, Couper and Conrad (2017), a crossover experiment in a probability web panel published in Public Opinion Quarterly, had the same respondents answer on both smartphone and PC. Despite smartphone users multitasking more, respondents "produced equally reliable data on both devices," and the authors conclude that people can give high-quality answers on a phone as long as they are presented with question formats that are easy to use on small touchscreens. In higher education specifically, Lambert and Miller (2015), "Living with Smartphones: Does Completion Device Affect Survey Responses?" in Research in Higher Education, similarly found device-related differences in completion behaviour and open-ended response length rather than wholesale distortion of substantive responses.

Synthesis: across a course-evaluation census (Champagne), a controlled mode experiment (Mavletova), a crossover design (Antoun et al.), and a higher-education survey study (Lambert & Miller), the pattern is consistent. Phones change participation and the length of what students write; they do not, on the current evidence, meaningfully bias the ratings themselves — and the open-text gap narrows sharply when the instrument is mobile-optimised.

Why it matters for course evaluation in practice

Four practical implications follow for QA and institutional-research teams.

1. Do not panic about your numbers — but watch your comments. The headline ratings appear robust to device. The genuine, evidence-backed risk is to qualitative feedback: a desktop-era form that is fiddly to type into on a phone will yield shorter, fewer comments. Since open text is the most actionable output, a mobile-hostile design silently degrades exactly the data teaching staff value most.

2. Mobile-first is a data-quality decision, not just a UX nicety. Antoun et al.'s result — equal reliability given touch-friendly formats — means the fix is concrete: large tap targets, single-question-per-screen flows, minimal horizontal scrolling, and avoidance of wide rating grids that collapse awkwardly on a 6-inch screen. Optimising for the phone is how you neutralise the open-text penalty.

3. Participation differences interact with non-response bias. If certain student groups disproportionately use phones and a non-optimised form depresses their completion, device effects can quietly become a representativeness problem. Monitoring completion by device is a cheap diagnostic.

4. Reconsider grid-heavy instruments. Wide Likert matrices are the format that travels worst to mobile. This reinforces the broader design lesson from the questionnaire-length and scale-format literatures: lean, single-construct, easy-to-tap items beat sprawling grids — on a phone the penalty for ignoring this is concrete and measurable.

Limitations and honest caveats

The evidence is encouraging but should not be over-read.

• Devices and people are confounded. Students who choose phones may differ from those who choose laptops (timing, context, conscientiousness). Champagne's and Lambert & Miller's observational comparisons cannot fully separate "the device" from "the kind of student or moment that uses the device." Antoun et al.'s crossover design addresses this — but in a general panel, not a teaching-evaluation setting.
• The technology moves. Some early mobile penalties (Mavletova 2013) reflect the smartphones and survey software of the early 2010s. Screens, browsers, and responsive design have improved substantially since, plausibly shrinking device gaps further — but also meaning older effect sizes may not describe today's devices precisely.
• "No difference in ratings" is not "no difference ever." Champagne found stable means; that does not rule out subtler effects on variance, on specific item types, or on particular subpopulations. Absence of a significant average difference is reassuring, not dispositive.
• Open-text shortening may be partly genuine. A shorter phone comment is not always lower quality — some students are simply more concise on a phone. Length is an imperfect proxy for the informativeness of feedback.
• Generalisability across systems. Results depend heavily on how mobile-friendly a given platform is. A census from one vendor's system (Champagne) reflects that system's mobile design as much as any universal law of devices.

The defensible conclusion is directional and design-contingent: device choice mainly affects participation and comment length, the substantive ratings look stable, and a touch-optimised instrument is the lever that closes the remaining gaps.

How Koji incorporates this

Koji is an AI-native, AI-moderated course-evaluation platform built for the reality that most students now respond on a phone — and several mechanisms target precisely the device-related risks the research identifies.

• Conversational, mobile-native interaction. Koji's core interface is an AI-moderated conversational interview — a chat-like, one-thing-at-a-time flow that maps naturally onto a phone screen, rather than a wide desktop-era Likert grid. This is the "easy-to-use-on-small-touchscreens" format that Antoun, Couper and Conrad (2017) identify as the condition under which mobile data quality matches PC.
• Protecting open-text depth where the penalty lives. Because the documented mobile cost concentrates in shorter, fewer comments, Koji is designed to elicit qualitative depth conversationally: instead of asking a student to type a paragraph into a small box, the AI moderator asks a focused follow-up question, then another — turning what would be a one-word phone comment into a short exchange. This directly counters the open-text shortening that Champagne (2013) and Mavletova (2013) observe.
• Automatic thematic analysis that does not penalise brevity. Koji's thematic analysis of open responses is designed to extract themes across many short answers, so feedback collected in concise mobile bursts still aggregates into usable, programme-level insight rather than being discarded as "too short."
• Participation-aware design and reporting. Because device can interact with who responds, Koji's emphasis on low-friction, mobile-first completion is aimed at sustaining response rates across student groups, and its reporting is built to surface representativeness rather than assume it.
• Quality scoring over raw length. Koji scores responses for substantiveness rather than treating a long answer as automatically better, so a crisp, informative phone comment is valued for its content — addressing the caveat that comment length is an imperfect proxy for quality.

As always the framing is careful: Koji is designed to mitigate the device-related risks to participation and qualitative depth, not to claim devices never matter. The same AI-moderated, mobile-native interview engine powers product and customer research on the core platform at koji.so, where smartphone completion is now the norm rather than the exception.

Related Resources

• Online vs. Paper Course Evaluations: Do Lower Response Rates Mean Worse Data?
• Survey Fatigue: Why Over-Surveying Students Quietly Wrecks Your Response Rates
• What Actually Raises Course-Evaluation Response Rates? The Experimental Evidence
• Why Students Click Straight Down the Middle: Satisficing in Course Evaluations
• Selection Bias in Course Evaluations: What Goos and Salomons Found
• How Many Responses Do You Need for a Reliable Course Evaluation?

References

• Champagne, M. V. (2013). Student Use of Mobile Devices in Course Evaluation: A Longitudinal Study. Educational Research and Evaluation, 19(7), 636–646. https://doi.org/10.1080/13803611.2013.834618
• Mavletova, A. (2013). Data Quality in PC and Mobile Web Surveys. Social Science Computer Review, 31(6), 725–743. https://doi.org/10.1177/0894439313485201
• Antoun, C., Couper, M. P., & Conrad, F. G. (2017). Effects of Mobile versus PC Web on Survey Response Quality: A Crossover Experiment in a Probability Web Panel. Public Opinion Quarterly, 81(S1), 280–306. https://doi.org/10.1093/poq/nfw088
• Lambert, A. D., & Miller, A. L. (2015). Living with Smartphones: Does Completion Device Affect Survey Responses? Research in Higher Education, 56(2), 166–177. https://doi.org/10.1007/s11162-014-9354-7