Turning Several Quality Signals Into One Defensible Decision: The Analytic Hierarchy Process for Course Evaluation
The Analytic Hierarchy Process derives criterion weights from pairwise comparisons and checks their internal consistency, giving a transparent, auditable way to combine student ratings, peer observation and learning outcomes into one decision — but it is subjective, vulnerable to rank reversal, and its 1–9 scale and 0.10 consistency threshold are conventions, not laws.
Koji Education Team
Product
In brief
The Analytic Hierarchy Process (AHP) is a decision method for combining several criteria of different kinds — student ratings, peer observation, learning outcomes, workload — into a single weighted judgement. Instead of asking a committee to state weights directly (which people do badly), AHP asks them to compare criteria two at a time ("how much more important is demonstrated learning than the satisfaction score?"), then extracts a consistent set of weights from those pairwise judgements using the matrix's principal eigenvector, and reports a consistency ratio that flags incoherent judging. Its value for course evaluation is transparency: the weights behind a personnel or programme decision become explicit and auditable. Its dangers are equally clear — the judgements are subjective, the 1–9 comparison scale and the 0.10 consistency cut-off are conventions, and AHP can exhibit rank reversal, where adding an irrelevant alternative flips the order of the others.
What the research says
Saaty (1977, Journal of Mathematical Psychology 15(3):234–281) introduced the method. His insight was that humans compare two things more reliably than they assign absolute scores to many, so AHP builds a pairwise comparison matrix on a 1–9 ratio scale (1 = equal importance, 9 = extreme dominance), then derives a priority (weight) vector as the matrix's principal eigenvector. Because human judgements are rarely perfectly transitive, Saaty defined a consistency index and a consistency ratio (CR) — the index scaled against the value expected from random judgements — with the now-standard rule of thumb that CR below 0.10 is acceptable and above it the comparisons should be revisited. Saaty (2008, International Journal of Services Sciences 1(1):83–98) is the compact modern statement of the full procedure: structure the decision as a hierarchy (goal → criteria → alternatives), collect pairwise comparisons at each level, synthesise the weights down the hierarchy, and check consistency throughout.
The method did not go unchallenged. Dyer (1990, Management Science 36(3):249–258) delivered the influential critique: because AHP normalises alternatives relative to each other, it can produce rank reversal — introducing a new, even dominated, alternative can change the preference order of the existing ones, which violates a basic axiom of rational choice and, Dyer argued, means AHP lacks a firm theoretical foundation unless linked to multi-attribute utility theory. Saaty replied and later variants ("ideal-mode" synthesis) mitigate the problem, but the debate is real and a careful user must know it. This places AHP within the broader family of multi-criteria decision analysis (MCDA), alongside simple additive weighting (SMART), outranking methods, and utility approaches — each trading off simplicity, theoretical rigour, and how they handle the weighting problem.
Why it matters for course evaluation in practice
Serious evaluation decisions are never single-number decisions. A promotion committee weighs student ratings and peer observation and teaching-portfolio evidence and learning outcomes; a programme review weighs satisfaction against progression, employability and equity. The universal temptation is to average these into a composite with weights someone picked in a corridor — 50% student ratings because it is the number everyone has. AHP replaces that opaque guess with a defensible, documented procedure: the committee reasons through which criteria dominate which, the arithmetic turns those comparisons into weights, and the consistency ratio catches a panel whose judgements are internally contradictory. When a faculty member disputes an outcome, the institution can show exactly how each criterion was weighted and that the weighting was coherent — a far stronger position than "we averaged the scores".
AHP is distinct from the platform's other combination methods, and choosing the right one matters. Rank aggregation combines several rankings of the same instructors into a consensus order; AHP instead derives criterion weights and applies them to score alternatives. Adaptive comparative judgement uses pairwise comparisons too, but of the objects being judged (essays, portfolios) to build a measurement scale, not of the criteria used to judge them. Importance-performance analysis reads importance off a regression or a stated-importance question rather than deriving it from structured pairwise reasoning. AHP earns its place precisely when the weighting itself is contested and needs to be made explicit and consistent.
A worked example shows the discipline in action. Suppose a promotion committee weighs four criteria: student ratings, peer observation, teaching-portfolio evidence and demonstrated learning gains. Rather than declaring student ratings the default 50%, each panellist compares the criteria pairwise — judging, for instance, that demonstrated learning moderately dominates student ratings, and that peer observation and portfolio evidence are roughly equal. The eigenvector of those comparisons might yield weights of 0.42 for learning, 0.27 for student ratings, 0.17 for peer observation and 0.14 for the portfolio, with a consistency ratio of 0.06 — comfortably coherent. The committee now has weights it reasoned into, a coherence check it passed, and a documented record it can defend if a candidate later contests the outcome.
Limitations and honest caveats
The method is a discipline for judgement, not a source of objectivity, and a PhD reader will press on four points. The weights are subjective. AHP structures and checks expert opinion; it does not discover a true importance. Different panels produce different weights, and the method makes that variability visible rather than removing it. Rank reversal is a genuine flaw (Dyer 1990) — validate decisions against the ideal-mode variant, or avoid adding and removing alternatives mid-process. The 1–9 scale and CR < 0.10 threshold are conventions. They work well in practice but have no deep theoretical necessity, and treating CR = 0.09 as "valid" and 0.11 as "invalid" is false precision. Consistency is not correctness. A panel can be perfectly consistent and consistently wrong; a low CR means the judgements hang together, not that they are right. And the number of comparisons explodes — n criteria need n(n−1)/2 comparisons, so a ten-criterion hierarchy is 45 pairwise judgements per panellist, which invites fatigue and satisficing. Keep hierarchies shallow, use it where the weighting is genuinely contested, and report the weights and the CR alongside the decision.
How Koji incorporates this
Koji treats AHP as a governance tool for the moment a QA process must combine dimensions into a decision, not as something to run on every survey. Its structured-question engine already produces the multidimensional evidence AHP needs — separate, clean measures of clarity, feedback, workload, inclusion and self-reported learning rather than one blurred overall number — and Koji is designed to support a pairwise-comparison workflow in which a review panel weights those dimensions, with the consistency ratio computed and surfaced so an incoherent set of judgements is flagged before a decision is signed off. Because the resulting weights are stored with the decision, Koji makes the reasoning auditable: a dean can see that student ratings were weighted, say, 30% against 45% for demonstrated learning, and that the panel's comparisons were consistent. Koji frames AHP output as "designed to make the weighting explicit and internally consistent", never as an objective importance, and it deliberately keeps the composite separate from the raw dimension scores so a reader can always inspect the underlying evidence — consistent with the platform's bias-aware, triangulate-across-sources philosophy. The same weighting-and-consistency machinery is available in Koji's core research platform at koji.so wherever a team must fold several research signals into one prioritisation.
Frequently asked questions
How is AHP different from just assigning percentage weights to each criterion?
Direct weighting asks people to state numbers they are bad at estimating and never checks coherence. AHP derives the weights from easier pairwise comparisons and reports a consistency ratio, so it both elicits better judgements and flags when a panel contradicts itself.
What is the consistency ratio and what value is acceptable?
The consistency ratio scales how internally contradictory the pairwise comparisons are against the value expected from random judgements. Saaty's rule of thumb is that below 0.10 is acceptable; above it, revisit the comparisons. The threshold is a convention, not a hard law.
What is rank reversal and should I worry about it?
Rank reversal is when adding or removing an alternative changes the preference order of the others, which Dyer (1990) flagged as a theoretical weakness. It matters if your decision set changes mid-process; using the ideal-mode variant of AHP and fixing the alternative set reduces the risk.
How many pairwise comparisons will a committee have to make?
For n criteria it is n(n−1)/2. Five criteria need ten comparisons; ten criteria need forty-five. Keep the hierarchy shallow and the criterion list short to avoid fatigue and careless judging.
Is AHP the same as adaptive comparative judgement?
No. Adaptive comparative judgement compares the objects being assessed (such as student work) to build a measurement scale. AHP compares the criteria used to make a decision to derive their weights. They share the pairwise idea but answer different questions.
Does a low consistency ratio mean the decision is correct?
No. Consistency means the judgements hang together logically; it says nothing about whether the panel's values are right. A consistent panel can still weight the wrong things. Consistency is necessary hygiene, not validity.
Related Resources
- Combining Several Rankings Into One Fair Order: Rank Aggregation
- Importance-Performance Analysis: Turning Scores into a Priority Map
- Adaptive Comparative Judgement in Evaluation
- The Dimensionality Debate and Using Ratings for Personnel Decisions
- Why Course-Evaluation Scores Misclassify Instructors
References
- Saaty, T. L. (1977). A scaling method for priorities in hierarchical structures. Journal of Mathematical Psychology, 15(3), 234–281. https://doi.org/10.1016/0022-2496(77)90033-5
- Saaty, T. L. (2008). Decision making with the analytic hierarchy process. International Journal of Services Sciences, 1(1), 83–98. https://doi.org/10.1504/IJSSCI.2008.017590
- Dyer, J. S. (1990). Remarks on the analytic hierarchy process. Management Science, 36(3), 249–258. https://doi.org/10.1287/mnsc.36.3.249
Related articles
Even Fair Evaluations Can Rank Unfairly: Why Course-Evaluation Scores Misclassify Instructors
A simulation-based finding that even unbiased, reliable, and valid course-evaluation scores produce high misclassification rates when used to rank or compare instructors — and what it means for tenure, promotion, and merit decisions.
Importance-Performance Analysis: Turning Course-Evaluation Scores into a Priority Map
A course-evaluation report that lists twenty item means tells you nothing about where to act first. Importance-Performance Analysis (IPA) plots each attribute by how much it matters to students against how well you did, producing a four-quadrant map that separates urgent fixes from wasted effort.
Combining Several Rankings Into One Fair Order: Rank Aggregation for Course Evaluation
When you have to merge several rankings - by different criteria, cohorts or panel members - into one, the method you pick changes the winner. Kemeny, Borda and Condorcet from social-choice theory show why, and how to do it defensibly.
One Number or Many? The Dimensionality Debate and How to Use Student Ratings for Personnel Decisions
Should a promotion committee use a single global teaching score or a detailed profile of many dimensions? The 1997 d'Apollonia & Abrami vs Marsh & Roche debate set the terms — and the answer depends on whether the purpose is summative judgement or formative improvement.