Response Styles and Likert Scales: Why Cross-Cultural Evaluation Needs More Than Numbers

In brief: How students use a rating scale — their tendency to agree regardless of content (acquiescence) or to pick the extremes (extreme response style) — varies systematically by culture and country. In Europe's multinational, multilingual classrooms this means a raw Likert average can conflate genuine satisfaction with response style, making cross-cohort comparisons unreliable. The evidence (Harzing, 2006; Baumgartner & Steenkamp, 2001; Van Vaerenbergh & Thomas, 2013) implies evaluation should lean less on bare scale means and more on reasoned, triangulated evidence.

The question, stated precisely

A European university evaluating a programme with German, Spanish, Italian, Chinese, and Nigerian students is making a quietly heroic assumption every time it averages a Likert item: that a "4 out of 5" means the same thing to each of them. Response-style research says it does not. Independently of what they actually think about a course, respondents from different cultures differ in how they use the scale itself — some gravitate to the endpoints, some to the middle, some agree with almost any statement put to them.

The practical question for quality assurance is therefore: how much of the variation in evaluation scores across nationalities, exchange cohorts, or international branch campuses reflects real differences in the student experience, and how much is an artefact of culturally patterned scale use? If the second component is large, league tables and cross-programme comparisons built on raw means are measuring partly the wrong thing.

What the research says

The anchor is Harzing (2006), "Response Styles in Cross-national Survey Research: A 26-Country Study" (International Journal of Cross Cultural Management, 6(2), 243–266). Across 26 countries, Harzing documented major differences in response styles between nations, and showed that country-level cultural characteristics — power distance, collectivism, uncertainty avoidance, and extraversion — significantly predict acquiescence and extreme response styles (ERS). Extreme response style is the tendency to mark the endpoints (e.g., "strongly agree") regardless of item content; acquiescence is the tendency to agree more than disagree. Harzing found, for instance, that response styles track collectivistic values such as embeddedness and traditionalism, and that extraversion is positively associated with ERS. The headline implication is unambiguous: nationality is confounded with scale use, so comparing raw means across countries compares apples with culturally calibrated oranges.

This is reinforced for the European context specifically by Baumgartner & Steenkamp (2001), "Response Styles in Marketing Research: A Cross-National Investigation" (Journal of Marketing Research, 38(2), 143–156). Using representative consumer samples from 11 European Union countries, they examined five forms of stylistic responding — acquiescence, disacquiescence, extreme response style/response range, midpoint responding, and noncontingent responding — and showed they exert systematic effects on scale scores, with the distortion depending on scale features such as the proportion of reverse-scored items and how far the scale mean sits from the midpoint. Crucially, they demonstrated that correlations between scales can be biased upward or downward by response style, meaning the problem is not just shifted averages but distorted relationships — exactly the relationships a QA analyst relies on when, say, linking "teaching clarity" to "overall satisfaction."

The broader methodological state of play is summarised by Van Vaerenbergh & Thomas (2013), "Response Styles in Survey Research: A Literature Review of Antecedents, Consequences, and Remedies" (International Journal of Public Opinion Research, 25(2), 195–217). They define response styles as the tendency to respond to items in a particular way regardless of content, catalogue their antecedents and consequences, and — importantly — review the available remedies: balanced scales with reverse-scored items, ipsative/forced-choice formats, statistical corrections, and richer question design. Their review makes clear that response styles are a recognised, correctable source of systematic error, not an exotic edge case.

Together the three establish a robust conclusion: scale use is culturally patterned, it materially distorts both means and correlations, and naive averaging across heterogeneous cohorts is a measurement error, not a neutral default.

Why it matters for course evaluation in practice

For European higher education specifically, the stakes are high:

• International cohorts are the norm, not the exception. Erasmus mobility, international master's programmes, and branch campuses mean most evaluation datasets mix nationalities. Harzing's findings imply that ranking these cohorts or instructors on raw Likert means embeds a cultural artefact.
• Cross-programme benchmarking is vulnerable. A programme with a higher proportion of high-acquiescence or high-ERS nationalities can post better scores without delivering better teaching — a confound that disproportionately affects exactly the internationalised programmes universities most want to assess fairly.
• Distorted correlations mislead improvement. Because response style can bias inter-scale correlations (Baumgartner & Steenkamp), a QA analyst trying to identify the drivers of satisfaction may chase relationships that are partly statistical artefacts.
• Language compounds the problem. Translated instruments add measurement-equivalence concerns on top of response style, so the assumption of comparability is doubly fragile in multilingual settings.

The defensible response is to reduce dependence on the bare scale mean and to anchor judgements in reasoned, content-rich evidence that response style cannot so easily distort.

Limitations and honest caveats

A careful reader should temper the implications:

• Response style is not the only — or always the largest — source of variance. Genuine differences in teaching quality and student experience are real; over-attributing cross-cohort gaps to response style would be its own error. The point is to separate the components, not to dismiss all cross-cultural differences as artefact.
• Country is a crude proxy for culture. Harzing's country-level associations describe averages; individuals within any nationality vary enormously, and treating nationality as destiny risks stereotyping. Individual-level acquiescence and ERS are better modelled directly than inferred from passport.
• Corrections carry assumptions. Statistical remedies (e.g., standardising, modelling ERS latent factors) assume a model of how style operates; a mis-specified correction can introduce new bias. Balanced scales and forced-choice formats have their own trade-offs in respondent burden and interpretability.
• Much foundational evidence is from marketing and general survey contexts. Baumgartner and Steenkamp studied consumers, not students; the direction of the effect is well established, but the precise magnitude in course-evaluation settings is less directly measured.

Naming these limits is what separates principled adjustment from mechanical correction.

How Koji incorporates this

Koji for Education is designed to lessen the grip that culturally patterned scale use has on evaluation conclusions:

• It reduces the weight carried by any single Likert number. Because Koji's AI-moderated interview captures reasoned explanation alongside ratings, an evaluator is not forced to treat a bare "4" or "5" as the whole signal — and reasoned narrative is far less susceptible to acquiescence and extreme-response patterning than an isolated scale point.
• It uses question formats that blunt response style. Beyond scale items, Koji supports ranking, single_choice, multiple_choice, and yes_no, plus open-ended probes. Forced-choice and ranking formats are among the remedies Van Vaerenbergh and Thomas highlight precisely because they constrain acquiescence and midpoint/extreme tendencies.
• It validates extreme and acquiescent answers conversationally. When a student rates everything at the ceiling, Koji can probe — "what specifically made it a 5?" — surfacing whether the rating reflects genuine enthusiasm or a stylistic tendency to pick the endpoint, the exact ERS pattern Harzing quantified.
• Its thematic analysis of open text is response-style-agnostic. Themes extracted from what students actually say do not inherit the acquiescence or extreme-response bias that contaminates numeric scales, giving cross-cultural cohorts a comparison basis that does not depend on identical scale calibration.
• Bias-aware, triangulated reporting keeps cohort composition visible and compares qualitative themes across nationalities rather than collapsing everything into one culturally confounded mean.

Koji is built to mitigate response-style distortion, not to claim it is eliminated — no instrument fully escapes how people use scales — but shifting weight from bare numbers to reasoned, triangulated evidence is exactly the direction the literature recommends. The same AI-moderated interview engine powers Koji's core research platform at koji.so, where comparing feedback fairly across international customer segments poses the identical challenge.

Related resources

• Why Averaging Likert Scores Misleads — the single-number problem this builds on
• What Can Open-Text Student Comments Tell You That Likert Scores Cannot?
• Do Student Evaluations Measure Learning? The Uttl Meta-Analysis Revisited
• Gender Bias in Student Evaluations of Teaching: Evidence and Mitigation
• Turning Student Feedback into ESG / ENQA Accreditation Evidence

Practical guidance for evaluation committees

When cohorts span many nationalities, committees can take several concrete steps to keep response style from contaminating conclusions. Avoid ranking instructors or programmes on raw Likert means whenever cohort composition differs markedly, because Harzing's evidence implies such rankings partly measure culturally patterned scale use rather than teaching quality. Prefer balanced instruments that mix positively and negatively worded items, since the distortion documented by Baumgartner and Steenkamp depends heavily on the proportion of reverse-scored items. Supplement scale questions with ranking or forced-choice formats, which Van Vaerenbergh and Thomas list among the most effective remedies for acquiescence and extreme responding. Where statistical corrections are used, model response style at the individual level rather than inferring it from nationality, both for accuracy and to avoid stereotyping diverse students by passport. Above all, shift interpretive weight toward reasoned, content-rich evidence — the explanations behind a rating — which is far less susceptible to stylistic patterning than an isolated scale point. The goal is not to deny real cross-cultural differences in experience but to separate them from differences in how a five-point scale is used.

References

• Harzing, A.-W. (2006). Response styles in cross-national survey research: A 26-country study. International Journal of Cross Cultural Management, 6(2), 243–266. https://doi.org/10.1177/1470595806066332
• Baumgartner, H., & Steenkamp, J.-B. E. M. (2001). Response styles in marketing research: A cross-national investigation. Journal of Marketing Research, 38(2), 143–156. https://doi.org/10.1509/jmkr.38.2.143.18840
• Van Vaerenbergh, Y., & Thomas, T. D. (2013). Response styles in survey research: A literature review of antecedents, consequences, and remedies. International Journal of Public Opinion Research, 25(2), 195–217. https://doi.org/10.1093/ijpor/eds021