The Success Case Method: Evaluating a Course Through Its Best and Worst Cases

The short answer

The Success Case Method (SCM), developed by Robert Brinkerhoff, evaluates a programme by deliberately ignoring the average and studying the extremes: the students who got the most out of a course and the students who got the least. It asks a sharper pair of questions than a satisfaction survey — when this course works, how well does it work and what makes the difference? and when it fails, what got in the way? You run a brief whole-population screen to locate the high and low performers, then interview a purposive sample of each in depth and verify their claimed outcomes. The result is not a mean but a set of evidenced stories that show the range of impact and its causes. It is one of the most useful methods for a course whose average score hides a badly split experience.

Bottom line: SCM trades representativeness for explanatory power. It cannot tell you the average student''s experience, but it can tell you what makes your course succeed or fail — and it produces verified, quotable evidence that a mean of 4.1 never will. Use it when the distribution matters more than the midpoint.

What the research says

Brinkerhoff set out the method in The Success Case Method: Find Out Quickly What''s Working and What''s Not (2003) and formalised it in Brinkerhoff (2005), The Success Case Method: A Strategic Evaluation Approach to Increasing the Value and Effect of Training (Advances in Developing Human Resources, 7(1), 86–101). The design has four moves. First, build an impact model — an explicit statement of what success from the course should look like (what students should be able to do, and what that should enable). Second, screen the whole population with a short survey to identify who appears to be at the extremes of success and non-success. Third, interview a purposive sample of the extreme cases in depth. Fourth — the step that separates SCM from mere storytelling — verify the claimed successes: the evaluator probes for concrete, checkable evidence that the outcome actually happened and was plausibly linked to the course, and equally documents what blocked the non-successful cases.

The methodological foundation is extreme-case (or deviant-case) sampling, described in the qualitative-evaluation tradition by Patton (2015). Patton''s argument is that the extremes are often more informative than the average, because the mechanisms that drive an outcome are most visible where the outcome is strongest and weakest. The mean tells you whether; the extremes tell you why. Brinkerhoff''s contribution was to package this into a fast, decision-oriented protocol and to insist on verification so that the persuasive power of a success story is backed by evidence rather than by selection.

The approach shares DNA with several methods a course evaluator may already know. Like realist evaluation, it seeks the context–mechanism–outcome logic of what works for whom. Like the critical incident technique, it works from concrete episodes rather than global ratings. Like the Most Significant Change technique, it takes stories seriously as data. Its distinctive move is the deliberate, two-tailed focus on both the best and the worst cases, and the requirement to verify impact.

Why it matters for course evaluation in practice

The single most common failure in course evaluation is reporting a mean over a distribution that is not unimodal. A course can score 3.5 because every student found it mediocre, or because half found it transformative and half found it useless — and those two courses need completely different interventions. The average cannot tell them apart. SCM is built precisely for the second case.

• It surfaces mechanisms, not just levels. Interviewing the students who thrived reveals what the course did right for them (a particular assignment, a support structure, a prerequisite they happened to have); interviewing those who struggled reveals the barriers. That is a redesign brief, not just a score.
• It produces credible evidence for stakeholders. Deans, external examiners and accreditation panels respond to verified stories of impact far more than to a decimal. A documented case — "this student could not do X at intake, can demonstrably do it now, and attributes it to these course features" — is exactly the kind of evidence-of-learning narrative that quality frameworks reward, provided the attribution is handled honestly.
• It is efficient. You do not deep-interview everyone; the whole-population screen is light, and the costly qualitative work is concentrated on the ~10–15% of cases at each tail where the learning is densest.
• It complements variance-hiding metrics. Run alongside your standard ratings, SCM answers the question the ratings raise but cannot resolve: why is this course working brilliantly for some and failing others?

Limitations and honest caveats

• Positive-story bias is the central risk. Success stories are seductive, and an evaluator (or an institution) keen to show impact can over-weight them. Brinkerhoff''s verification step is the designed defence, but it is only as good as the rigour of the probing; a poorly run SCM becomes a marketing exercise.
• It is not representative — by design. Extreme cases cannot estimate population averages or the prevalence of success. The screening survey gives a rough base rate, but SCM should never be reported as if the vivid cases were typical.
• Attribution remains inferential. Linking a student outcome to the course, rather than to prior ability, motivation or a parallel experience, is a judgement. This is where SCM should borrow from contribution analysis: build and test the causal claim, do not assert it.
• Self-report and recall. The interviews rely on students'' accounts of what they gained and why, with all the reconstruction and self-presentation that entails.
• The impact model can be wrong. If the definition of "success" baked into step one is narrow or mistaken, the whole study inherits the error. Garbage in, evidenced garbage out.
• Small qualitative samples. Generalisation beyond the studied cohort and context should be cautious.

None of these sink the method; they define how to run it honestly — verify claims, report the base rate, be explicit about attribution, and pair it with quantitative coverage.

How Koji incorporates this

SCM''s two-stage design — a light screen of everyone, then deep verified interviews of the extremes — maps almost exactly onto how Koji works, which is why the method is unusually well suited to an AI-moderated platform.

• Automated screen-then-interview. Koji can field the whole-population screening study with scale and single_choice items, identify the high and low tails, and route those students into deeper AI-moderated conversational interviews — the SCM structure, run without an evaluator having to schedule dozens of interviews by hand.
• Verification probing built in. The positive-story-bias problem is a probing problem, and probing is what Koji''s interview engine does: when a student claims a gain, the AI can ask for a concrete, checkable example and for the specific course features they credit — operationalising Brinkerhoff''s verification step and mitigating (not removing) the bias toward flattering narratives.
• Two-tailed by intent. Koji interviews the non-successful cases with the same rigour as the successes, surfacing the barriers a satisfaction survey never captures, and its thematic analysis extracts the enabling and blocking factors across both tails.
• Report generation for stakeholders. Koji can assemble verified cases with representative quotations into a report an evaluation committee or accreditation panel can audit — evidence, not anecdote.
• Honest boundaries. Koji runs the mechanics; the impact model, the attribution judgement, and the decision about how much weight the extreme cases deserve remain human. The method''s limitations do not disappear because it is automated — Koji''s reporting keeps the screening base rate visible so vivid cases are never mistaken for typical ones.

The same pattern powers Koji''s core research platform at koji.so, where product teams interview their power users and their churned users — and skip the indifferent middle — to learn what actually drives and blocks value.

Running an SCM study step by step

In practice a course-level Success Case study runs as five concrete steps. First, write the impact model: state, before collecting anything, what a student who succeeded in this course should be able to do and what that should enable downstream — this is the yardstick everything else is measured against, and getting it wrong contaminates the whole study. Second, screen the whole cohort with a short survey (a few scale items plus one or two behavioural questions) to locate the apparent high and low performers; keep it light, because its only job is to sort, not to explain. Third, sample the tails purposively — typically the top and bottom 10–15% — rather than a random cross-section. Fourth, interview in depth and verify: for each claimed success, press for a concrete example and checkable evidence, and for each struggle, document the specific barrier; this verification step is what separates evidence from anecdote. Fifth, synthesise and report the range — present the success mechanisms, the failure barriers, and, crucially, the base rate from the screen so readers know how common each pattern is. Reported this way, an SCM study gives a programme committee something a satisfaction mean never can: a causal account of why the course works brilliantly for some students and fails others, backed by evidence they can audit.

Related resources

• The Most Significant Change Technique: Story-Based Course Evaluation
• The Critical Incident Technique for Course Feedback
• Contribution Analysis: Making Credible Causal Claims from Course Evaluation
• Realist Evaluation for Course Feedback: What Works, for Whom
• Designing Course Evaluation for Use: The Utilization-Focused Approach
• Developmental Evaluation: Evaluating Course Innovation Under Uncertainty

References

• Brinkerhoff, R. O. (2005). The Success Case Method: A strategic evaluation approach to increasing the value and effect of training. Advances in Developing Human Resources, 7(1), 86–101. https://doi.org/10.1177/1523422304272172
• Brinkerhoff, R. O. (2003). The Success Case Method: Find Out Quickly What''s Working and What''s Not. San Francisco: Berrett-Koehler.
• Patton, M. Q. (2015). Qualitative Research & Evaluation Methods (4th ed.). Thousand Oaks, CA: Sage. (Extreme/deviant-case purposive sampling.)
• Coryn, C. L. S., Schröter, D. C., & Hanssen, C. E. (2009). Adding a time-series design element to the Success Case Method to improve methodological rigor. American Journal of Evaluation, 30(1), 80–92. https://doi.org/10.1177/1098214008326557