Goal-Free Evaluation: Why Scriven Says Course Reviews Should Ignore the Stated Objectives

In brief

Goal-free evaluation, developed by the philosopher-evaluator Michael Scriven, deliberately keeps the evaluator ignorant of a course's stated goals and objectives, judging it instead on the actual effects it produces measured against the needs of the students it serves. The point is to escape "goal-related tunnel vision" — the tendency to look only where success was promised — and to give unintended side effects the same standing as intended ones. For course review, it is a disciplined way to ask a question conventional surveys almost never ask: what is this course actually doing, including the things nobody designed it to do?

What the research says

Scriven introduced goal-free evaluation in the early 1970s and returned to it repeatedly, most accessibly in Prose and Cons about Goal-Free Evaluation (1991). His argument has two prongs. The first is about bias: an evaluator who knows the intended objectives is primed to search for evidence of those objectives and to treat everything else as peripheral. Goal-free evaluation removes the goals from the evaluator's field of view — classically by commissioning an external evaluator who is intentionally screened from the programme's aims — so that judgement starts from observed effects, not promised ones.

The second prong is about side effects. After several projects in which the unplanned consequences mattered more than the official targets, Scriven concluded that the very distinction between "intended" and "unintended" effects is arbitrary from the standpoint of the people affected. A course designed to teach statistical technique may, as a side effect, destroy or ignite students' confidence to use quantitative methods at all — an outcome that dwarfs the syllabus in importance but that no objectives-derived questionnaire will capture. In goal-free mode the evaluator profiles the needs of those served and asks whether the course meets them, whatever the brochure claimed.

Goal-free evaluation belongs to Scriven's broader contribution to evaluation theory, including his foundational 1967 distinction between formative and summative evaluation. It stands in deliberate contrast to the objectives-based tradition associated with Ralph Tyler, in which success is defined precisely as the attainment of pre-stated objectives. It also resonates with Lee Cronbach's (1963) argument that evaluation should serve course improvement by illuminating what actually happens, not merely certify goal attainment.

The empirical case for taking side effects seriously is now strong in the SET literature specifically. Deslauriers and colleagues (2019) showed that active-learning classes can lower students' feeling of learning even as they raise measured learning — a systematic, evaluation-relevant side effect that a satisfaction-and-objectives survey would misread as failure. More broadly, Uttl, White and Gonzalez (2017) found student ratings explain roughly 1% of variance in learning, and Stark and Freishtat (2014) catalogued how conventional reporting misleads. If the standard instrument is a weak proxy pointed only at intended targets, an approach that widens the aperture to actual, unplanned effects is a serious corrective rather than a curiosity.

Why it matters for course evaluation in practice

1. Catch the side effects that decide a programme's real value. Retention, confidence, identity as a "maths person" or "not a maths person," workload spillover onto other modules, assessment-driven surface learning — these are frequently the most consequential outcomes and are almost never on the objectives-derived form. A goal-free component is designed to surface them.

2. Reduce the confirmation bias baked into fixed item banks. A questionnaire assembled from a course's learning outcomes can only confirm or disconfirm those outcomes. It is structurally blind to the unexpected. Goal-free enquiry starts from the student's experience and works outward.

3. Strengthen summative defensibility. For high-stakes programme review and revalidation, a finding that survives even when the evaluator was blind to the programme's own goals is harder to dismiss as marking one's own homework.

Limitations and honest caveats

• True blindness is hard. In a small department everyone knows what a course is "for." Perfect screening is often impossible, so goal-free evaluation is better understood as a stance — hold the stated goals at arm's length — than an absolute condition.
• Needs assessment carries the weight. If goals do not define quality, needs must. But whose needs, defined how? A weak or contested needs profile simply swaps the programme's public objectives for the evaluator's private ones — which is the bias goal-free evaluation set out to avoid.
• It can be resource-intensive. A dedicated external, blind evaluator is a luxury few institutions can run for every module every term. In practice goal-free elements are grafted onto routine cycles rather than run in their pure form.
• It does not remove all subjectivity. Screening out goals shifts the locus of judgement; it does not eliminate it. Evaluator standards, framing, and the well-documented negativity bias in reading open comments still operate.
• Not a substitute for goal-based review. Accreditors expect evidence that stated learning outcomes were addressed. Goal-free evaluation complements that expectation; it cannot replace it.

How Koji incorporates this

The practical obstacle to goal-free evaluation has always been cost: it wants an open-minded, needs-focused enquiry into every course, and human evaluators do not scale that far. Koji's design closes much of that gap.

• Non-leading, open AI-moderated interviews. Koji's conversational interviewer opens from the student's experience with open_ended prompts that are not derived from the course's stated learning outcomes, so unanticipated effects have room to surface. This is goal-free enquiry operationalised at cohort scale, not a one-off external review.
• Automatic thematic analysis of side effects. Because the open text is analysed thematically rather than scored against a fixed rubric, positive and negative unplanned effects — confidence gained or lost, workload spillover, changed disciplinary identity — cluster into visible themes instead of vanishing between the lines of a Likert grid.
• Bias-aware moderation. The interview protocol is explicitly designed to avoid sycophantic or leading questions that would nudge students back toward the "official" story — a safeguard aligned with Scriven's core concern about goal-primed searching. (Koji frames this as mitigating leading-question bias, not eliminating it.)
• A needs-referenced backbone. Structured scale and single_choice items about what students actually needed from the course give the goal-free findings a defensible standard of quality, addressing the "whose needs?" objection above rather than leaving judgement free-floating.

Two honest boundaries. Koji does not pretend an AI can be perfectly "blind" to a course's purpose; rather, it holds stated objectives out of the question wording so they do not steer the conversation. And a goal-free layer is meant to sit alongside your outcomes-based evidence for accreditation, not to replace it. The same open-ended, non-leading interview engine drives Koji's core research platform at koji.so, where discovering the effect nobody designed for is the essence of good product research.

Designing a goal-free component into a routine cycle

Few institutions can commission a blind external evaluator for every module, but the goal-free stance can be built into an ordinary cycle cheaply. The practical recipe is to run two streams in parallel and keep them methodologically separate. The first is your conventional goal-based stream: items derived from the stated learning outcomes, reported against those outcomes for accreditation. The second is a goal-free stream whose prompts are deliberately written without reference to the syllabus — open questions about what students actually gained, lost, or noticed, analysed for whatever themes emerge rather than scored against a rubric.

The discipline that makes this work is sequencing and separation: the team analysing the goal-free stream should, as far as practical, interpret its themes before consulting the course's stated aims, so the aims cannot retro-fit the reading. When the two streams are finally compared, the interesting findings are precisely the mismatches — effects the goal-free stream surfaced that the goal-based stream had no item for. Those are the side effects Scriven cared about, and they are where the improvement value usually hides. Treat convergence between the streams as reassurance and divergence as the agenda for the next programme meeting. Run this way, goal-free evaluation costs little more than rewording a section of an existing survey, yet it recovers most of the diagnostic power that a fully independent, blind review would provide.

Related resources

• Open-ended course evaluation question design: specific prompts
• Utilization-focused evaluation: designing course feedback for use
• Student written comments in course evaluation
• Negativity bias in reading open-text comments
• Text analytics for open-ended student comments
• An evaluation of course evaluations (Stark & Freishtat)

References

• Scriven, M. (1991). Prose and cons about goal-free evaluation. Evaluation Practice, 12(1), 55-62. https://doi.org/10.1177/109821409101200108
• Scriven, M. (1967). The methodology of evaluation. In R. W. Tyler, R. M. Gagne & M. Scriven (Eds.), Perspectives of Curriculum Evaluation (pp. 39-83). Chicago: Rand McNally.
• Cronbach, L. J. (1963). Course improvement through evaluation. Teachers College Record, 64(8), 672-683.
• Deslauriers, L., McCarty, L. S., Miller, K., Callaghan, K. & Kestin, G. (2019). Measuring actual learning versus feeling of learning in response to being actively engaged in the classroom. Proceedings of the National Academy of Sciences, 116(39), 19251-19257. https://doi.org/10.1073/pnas.1821936116
• Uttl, B., White, C. A. & Gonzalez, D. W. (2017). Meta-analysis of faculty's teaching effectiveness: Student evaluation of teaching ratings and student learning are not related. Studies in Educational Evaluation, 54, 22-42. https://doi.org/10.1016/j.stueduc.2016.08.007
• Stark, P. B. & Freishtat, R. (2014). An evaluation of course evaluations. ScienceOpen Research. https://doi.org/10.14293/S2199-1006.1.SOR-EDU.AOFRQA.v1