A High Correlation Does Not Mean Two Evaluation Methods Agree: The Bland-Altman Limits of Agreement

In brief

Suppose you want to know whether a short AI-moderated evaluation gives the "same" result as your legacy paper form, or whether peer review agrees with student ratings. The instinct is to correlate the two and report a high r. That is the wrong test. A strong correlation only says the two methods rank courses similarly; it does not say they give the same number, and two methods can correlate at 0.95 while one runs a full point higher than the other. The Bland-Altman method — the standard in clinical measurement — plots the difference between the two methods against their average, reports the mean difference (bias) and the 95% limits of agreement, and answers the real question: can I swap one method for the other without changing my decisions?

What the research says

The foundational paper is J. Martin Bland and Douglas Altman, Statistical methods for assessing agreement between two methods of clinical measurement (Bland & Altman, 1986, The Lancet, 327(8476), 307–310, DOI 10.1016/S0140-6736(86)90837-8) — one of the most-cited papers in science, with tens of thousands of citations. Their argument is simple and devastating for the correlation habit: correlation measures association, not agreement. Two methods can be perfectly correlated yet systematically differ, because correlation is unchanged if you add a constant or multiply one method by a factor. Worse, correlation is inflated by a wide range of true values — a heterogeneous set of courses will produce a high r almost regardless of how well the methods agree.

Their alternative is the limits of agreement. For each unit measured by both methods, compute the difference. The mean of those differences is the bias (systematic offset); the standard deviation of the differences gives the 95% limits of agreement as bias ± 1.96·SD, the interval containing about 95% of differences. Plotting the difference against the average of the two methods (the Bland-Altman plot) then reveals whether disagreement is constant or grows with the score, and whether any courses are extreme outliers.

Bland and Altman extended the approach — including handling repeated measurements and non-constant variability — in Measuring agreement in method comparison studies (Bland & Altman, 1999, Statistical Methods in Medical Research, 8(2), 135–160, DOI 10.1177/096228029900800204). A clear modern tutorial is Davide Giavarina's Understanding Bland Altman analysis (Giavarina, 2015, Biochemia Medica, 25(2), 141–151, DOI 10.11613/BM.2015.015), which walks through construction, interpretation and the common mistakes — chiefly, that the limits of agreement are a description of how far apart the methods are, and whether that spread is acceptable is a judgement the analyst must make in advance, not read off the data.

This is a genuinely different question from the validity work already in the corpus. The multitrait-multimethod matrix asks whether methods converge on an underlying construct using correlations; Bland-Altman asks the narrower, more operational question of whether two methods produce interchangeable numbers on the same scale.

Why it matters for course evaluation in practice

Course-evaluation offices constantly compare methods and then treat them as swappable. Three recurring cases are exactly Bland-Altman problems.

• Mode and instrument migration. When a university moves from paper to online, or from a legacy Likert form to an AI-moderated conversational format, someone asks "do we still get the same scores?" A correlation will look reassuringly high. A Bland-Altman analysis might reveal that the new mode runs 0.3 points lower on average with limits of agreement spanning a full point — meaning an individual instructor's score could shift materially with the switch. That is decision-relevant in a way r is not, and it complements the design questions in our note on mode effects and social desirability.
• Peer review vs student ratings. Institutions increasingly triangulate. But if peer observation and student ratings are used interchangeably in a personnel file, the limits of agreement tell you how large a discrepancy is normal — and whether a gap on one instructor is genuinely unusual or within routine method spread. The same logic applies to instructor self-ratings vs student ratings.
• Human vs automated coding. When an AI codes open-text feedback and you want to trust it against human coders, agreement — not correlation — is the acceptance test, and Bland-Altman handles the continuous case (e.g. sentiment scores) alongside kappa-style measures for categorical coding.

In every case, the discipline Bland-Altman forces is to decide, before looking, how much disagreement you are willing to tolerate — a pre-specified "acceptable difference" — and then check whether the limits fall inside it.

Limitations and honest caveats

• You must set the acceptable limit in advance. Bland-Altman describes the spread; it does not tell you whether that spread is tolerable. That threshold is a substantive judgement (how much score movement changes a decision), and skipping it turns the plot into decoration.
• Sample size. The limits of agreement are estimated with uncertainty; Bland and Altman recommend a reasonable number of paired observations (they suggest on the order of 40+, and more for tight confidence intervals on the limits). Small pilots give unstable limits, and the confidence interval around the limits is frequently ignored.
• The two methods need a shared scale. Limits of agreement are most natural when both methods report on the same metric (e.g. both a 1–5 rating). Comparing a Likert mean to a fundamentally different index needs transformation and care.
• Proportional bias and non-normal differences. If disagreement grows with the score, or the differences are skewed (common with ceiling-heavy evaluation data), the simple ±1.96·SD limits mislead; the 1999 extensions or a regression-based approach are needed.
• Agreement is not validity. Two methods can agree beautifully and both be biased in the same direction. Bland-Altman answers "are they interchangeable", not "are they right" — for the latter you still need construct-validity evidence.

How Koji incorporates this

Koji is frequently the new method in a method-comparison study — the conversational, AI-moderated instrument being introduced alongside a legacy form — so agreement analysis is directly in scope.

• Paired, structured data for clean comparison. Because Koji captures responses in consistent scale and structured formats with respondent-level linkage, it is straightforward to assemble the paired observations a Bland-Altman analysis needs (same course measured by two methods).
• Bias-aware migration reporting. When an institution moves to Koji, its reporting is designed to present the difference and its spread against a legacy instrument, not just a correlation — surfacing systematic offset and the range of individual-course movement so committees are not blindsided by a mode shift.
• Triangulation without false interchangeability. Koji supports collecting student, peer and self inputs and is built to present them as distinct sources with their agreement quantified, rather than silently averaging methods that a limits-of-agreement check would show are not interchangeable.
• Continuous and categorical agreement. For open-text, Koji's automatic thematic analysis and quality scoring produce structured outputs that can be checked against human coders using agreement statistics appropriate to the data type.

Koji's core research platform at koji.so applies the same paired-data discipline to product and customer research, where teams routinely need to know whether a new survey instrument agrees with an old one before switching.

The honest framing: Koji does not decide your acceptable limit of agreement — that is your institution's call. What Koji provides is the clean paired data and difference-focused reporting that make a proper agreement analysis possible, instead of a reassuring but misleading correlation.

Frequently asked questions

Why is correlation the wrong test for method agreement?

Correlation measures whether two methods rank things similarly, not whether they give the same number. Two methods can correlate at 0.95 while one is consistently a point higher; adding a constant to one method does not change the correlation at all. Bland-Altman looks at the differences directly.

What exactly are the limits of agreement?

They are the mean difference between the two methods plus or minus 1.96 times the standard deviation of the differences — the interval that contains about 95% of the differences. If those limits are narrow enough that any difference inside them would not change your decision, the methods are interchangeable for your purpose.

How many paired observations do I need?

Enough for stable estimates of the mean and SD of differences; Bland and Altman suggest on the order of 40 or more, with larger samples giving tighter confidence intervals on the limits themselves.

Does good agreement mean both methods are valid?

No. Two methods can agree and both be biased in the same way. Bland-Altman tells you whether methods are interchangeable, not whether either measures the right thing.

Can I use this to validate AI-generated evaluation scores?

Yes, for continuous outputs such as sentiment or quality scores: compare the AI method against human coders using limits of agreement, having first decided how much difference you are willing to accept.

References

• Bland, J. M., & Altman, D. G. (1986). Statistical methods for assessing agreement between two methods of clinical measurement. The Lancet, 327(8476), 307–310. https://doi.org/10.1016/S0140-6736(86)90837-8
• Bland, J. M., & Altman, D. G. (1999). Measuring agreement in method comparison studies. Statistical Methods in Medical Research, 8(2), 135–160. https://doi.org/10.1177/096228029900800204
• Giavarina, D. (2015). Understanding Bland Altman analysis. Biochemia Medica, 25(2), 141–151. https://doi.org/10.11613/BM.2015.015

Related resources

• The Multitrait-Multimethod Test for Course Evaluation
• Peer Observation vs Student Evaluations: Convergent Validity
• Do Instructors and Students Agree? Self-Evaluation vs Student Ratings
• Inter-Rater Reliability and Thematic Analysis of Open-Text Feedback
• Can a Large Language Model Code Your Open-Text Feedback?
• Do Students' Written Comments Match Their Ratings? Concordance