Rank Aggregation for Course Evaluation: Kemeny, Borda and Condorcet

In brief

You often need to combine several rankings of the same courses or instructors into one: rank by overall rating, by learning gain, by inclusion, and by workload appropriateness, then merge; or merge the separate rankings of several review-panel members. The uncomfortable result from social-choice theory is that the method you choose to combine them changes who comes out on top. Borda count (average the rank positions) is simple but can pick a course that a majority would rank below another. The Condorcet principle (respect every pairwise majority) is fairer but can produce cycles. The Kemeny-Young method resolves this by finding the single consensus order that disagrees least with all the inputs. And Arrow's impossibility theorem proves no method satisfies every fairness property at once — so the honest move is to pick a method deliberately, disclose it, and check robustness.

What the research says

Rank aggregation is the problem of turning many orderings into one consensus ordering. It appears across fields under different names — social choice, the median-ranking problem, the consensus-ranking problem — and its core tensions were mapped centuries ago.

Borda (1781) proposed scoring each alternative by its summed or averaged rank position. It is transparent and is exactly what an institution does when it averages "rank on criterion A, rank on criterion B." But Borda is not Condorcet-consistent: it can crown an option that would lose a head-to-head majority vote against another, because a few extreme placements move an average that pairwise comparisons ignore.

Condorcet (1785) argued the winner should be whoever beats every other alternative in pairwise majority comparisons. The catch, which Condorcet himself found, is the Condorcet paradox: majorities can cycle — A beats B, B beats C, C beats A — so no consistent overall order exists from majority rule alone.

Kemeny (1959), "Mathematics without numbers" (Daedalus, 88(4), 577-591), gave the resolution used today: choose the consensus ranking that minimises the total Kendall-tau distance (the count of pairwise disagreements) to all the input rankings. Young and Levenglick (1978), "A consistent extension of Condorcet's election principle" (SIAM Journal on Applied Mathematics, 35(2), 285-300, doi:10.1137/0135023), proved the Kemeny rule is the unique method that is neutral, consistent, and Condorcet — the strongest axiomatic justification available. Its price is computational: finding the exact Kemeny order is NP-hard even for four input rankings (Dwork, Kumar, Naor and Sivakumar, 2001, "Rank aggregation methods for the web," Proceedings of WWW10, 613-622, doi:10.1145/371920.372165), so in practice you use heuristics or the Borda order as a starting approximation.

Overarching all of it, Arrow (1951/1963), Social Choice and Individual Values, proved that no aggregation rule combining three or more alternatives can simultaneously satisfy a short list of reasonable fairness conditions (unrestricted domain, unanimity, independence of irrelevant alternatives, and non-dictatorship). There is no perfect method — only defensible choices with disclosed trade-offs.

Why it matters for course evaluation in practice

The moment you build a "top courses" or "teaching excellence shortlist" from more than one criterion, or pool several assessors' orderings, you are doing rank aggregation whether you name it or not — and the default (average the ranks) is Borda, with all its known weaknesses.

Averaging ranks can mislead. A course ranked 1st on satisfaction but 20th on workload-appropriateness averages to a middling position, while a course consistently 5th-6th on everything may top the average — even though most pairwise comparisons favour the specialist. Deciding whether that is the outcome you want is a values choice you should make openly.
Ties and cycles are informative. A Condorcet cycle among three instructors is not a nuisance to be broken silently; it is evidence that the criteria genuinely conflict and that a single ranking is imposing false precision.
Rank aggregation is not score aggregation. Working with ranks deliberately discards the size of differences, which protects against a few extreme scores dominating — a cousin of the concern behind empirical-Bayes shrinkage — but also throws away information that best-worst scaling or a proper scale would keep. Choose ranks when the scores are not comparable across criteria; keep scores when they are.

Above all, rank aggregation should make you humble about league tables. Fair-ranking research already shows that even a single well-measured score misclassifies instructors, and that ranking instructors by evaluation scores is often indefensible. Aggregating several noisy rankings compounds, rather than cancels, that fragility.

Limitations and honest caveats

Arrow's theorem is a hard limit. You cannot escape the trade-offs by finding a cleverer rule; you can only pick which property to sacrifice and say so.
Kemeny is NP-hard. Exact solutions are infeasible beyond small sets, and different heuristics can return different orders, so report the method and its approximation.
Rankings hide magnitude. A course ranked 1st might be a whisker or a mile ahead of 2nd. If the gaps matter, aggregate scores (with uncertainty) rather than ranks — and remember the 4.2 vs 4.4 trap: tiny score gaps become spurious rank gaps.
Garbage in, consensus out. Aggregation cannot repair biased or unreliable input rankings; if the underlying criteria are contaminated by grading leniency or charisma, the consensus inherits the bias.
Manipulability. Most rules can be gamed by strategic ordering; where stakes are high (promotion, funding), that matters.
Small numbers of rankers. With only two or three criteria or panellists, "consensus" is a strong word for what is really a coin-toss among near-ties.

How Koji incorporates this

Koji supports genuine ranking data and treats combining rankings as a deliberate, disclosed choice rather than a silent average.

Native ranking questions: Koji's ranking question type collects true ordinal preferences from students (for example, ranking what most helped their learning), so aggregation starts from real rankings, not from ranks manufactured by sorting means.
Method transparency: when Koji reporting combines multiple criteria or cohorts, the aggregation logic is explicit and the size of differences is shown alongside the order, so a hair's-breadth gap is not dressed up as a decisive lead.
Uncertainty preserved: because Koji reports intervals and flags near-ties, users can see when a consensus ranking is really a cluster of statistically indistinguishable options — the honest counterpart to a false league table.
Qualitative reconciliation of conflicts: when criteria genuinely cycle, the AI-moderated interview probes why students trade off, say, challenge against clarity, turning an unresolvable ranking into an explanation leaders can act on. Koji's core research platform at koji.so uses the same ranking and trade-off tooling for prioritising product features across segments.

Koji is designed to mitigate the false precision of merged league tables, not to declare one aggregation rule universally correct — the method must be chosen and justified for the decision at hand.

Frequently asked questions

What is rank aggregation?

It is the problem of combining several rankings of the same set of options — by different criteria, cohorts, or panel members — into a single consensus ranking. Social-choice theory shows the combining method itself affects the result.

Why not just average the rank positions?

Averaging ranks is the Borda count. It is simple but can select an option that a majority would rank below another in head-to-head comparisons, because averages are moved by a few extreme placements that pairwise comparisons ignore. It is a legitimate choice, but you should know its weakness.

What is a Condorcet cycle and why does it matter?

A Condorcet cycle is when majorities are intransitive: A beats B, B beats C, and C beats A. It means no single consistent ranking follows from pairwise majorities, which is a signal that your criteria genuinely conflict rather than a technical glitch to hide.

What makes the Kemeny-Young method special?

It selects the consensus ranking that minimises total pairwise disagreement with all inputs, and Young and Levenglick (1978) proved it is the unique rule that is neutral, consistent and Condorcet. The drawback is that computing it exactly is NP-hard, so heuristics are used for larger sets.

Does Arrow's theorem mean ranking is pointless?

No. It means no rule satisfies every fairness property at once, so there is no perfect method — only defensible ones. The practical response is to choose a method deliberately, disclose it, and test whether the conclusion is robust to that choice.

When should I aggregate scores instead of ranks?

Aggregate scores when the underlying measurements are comparable and the size of differences matters, because ranks discard magnitude. Use ranks when scores are not comparable across criteria, or when you deliberately want to limit the influence of a few extreme values.

Related resources

References

Kemeny, J. G. (1959). Mathematics without numbers. Daedalus, 88(4), 577-591.
Young, H. P., & Levenglick, A. (1978). A consistent extension of Condorcet's election principle. SIAM Journal on Applied Mathematics, 35(2), 285-300. doi:10.1137/0135023
Dwork, C., Kumar, R., Naor, M., & Sivakumar, D. (2001). Rank aggregation methods for the web. Proceedings of the 10th International Conference on World Wide Web (WWW10), 613-622. doi:10.1145/371920.372165
Arrow, K. J. (1963). Social Choice and Individual Values (2nd ed.). New Haven: Yale University Press. (Original work published 1951.)
de Borda, J. C. (1781). Memoire sur les elections au scrutin. Histoire de l'Academie Royale des Sciences, Paris.

Combining Several Rankings Into One Fair Order: Rank Aggregation for Course Evaluation