Let Students Name Their Own Criteria: The Repertory Grid Technique for Course Evaluation

In brief

A standard course-evaluation questionnaire imposes the institution''s categories on students: it asks about "clarity," "organisation," "workload," and "feedback" because those are the constructs the designers chose. But what if the dimension a student actually uses to judge the course — "whether the lecturer treated us as adults," "whether the assignments felt real," "whether I could ask a stupid question" — never appears on the form? The repertory grid technique, developed from George Kelly''s (1955) personal construct theory, is a structured method for eliciting the constructs students themselves use to make sense of their learning, rather than the ones we assume. It combines the openness of a qualitative interview with a matrix structure that can be analysed systematically. Used alongside standard surveys, it surfaces evaluation criteria a fixed questionnaire is blind to — and it does so in the students'' own language.

What the research says

Kelly''s personal construct theory (1955) begins from a simple premise: people make sense of the world through their own bipolar constructs — contrasts like fair–unfair, engaging–boring, rushed–well-paced — and no two people''s construct systems are identical. Because a pre-written questionnaire can only ask about the designer''s constructs, it systematically misses the ones respondents actually use. The repertory grid is Kelly''s instrument for eliciting a person''s own constructs in a structured, analysable form.

The technique has three parts (Fransella, Bell & Bannister, 2004; Jankowicz, 2004):

1. Elements — the concrete things being judged. For course evaluation these might be several courses the student has taken, several teaching sessions, or several activities within one module.
2. Constructs — elicited, not supplied. The classic method presents the student with three elements (a triad) and asks: "In what important way are two of these alike and different from the third?" The answer ("these two let me work things out myself, that one just told me the answer") yields a bipolar construct in the student''s own words. Repeating with different triads builds up the student''s personal set of dimensions.
3. The grid — the student then rates every element on every elicited construct, producing a matrix that can be analysed for structure (which constructs cluster, how the student''s criteria relate to one another).

Tarah Wright (2003), in Teaching in Higher Education, applied exactly this to undergraduate science courses, arguing that an adapted repertory grid works both as a research tool and as a classroom assessment technique — a practical way for university teachers to discover the criteria their own students bring to bear. Earlier work in higher education used repertory grids to elicit staff and students'' personal constructs of research, teaching, and professional development, documenting developmental shifts in how learners construe "good" work that a fixed scale would never have detected.

The methodological appeal is that the grid bridges qualitative and quantitative: the construct elicitation is open and phenomenological (it respects the student''s own meaning), while the resulting ratings matrix supports systematic analysis — clustering, comparison across respondents, and tracking change over time. It shares this hybrid character, and some analytic machinery, with concept mapping and with Q-methodology, but it is distinctive in eliciting the dimensions rather than supplying them.

Why it matters for course evaluation in practice

Most evaluation instruments have a construct-coverage problem they cannot see. If students judge a course partly on a dimension the survey omits, that signal is invisible — not low, not high, simply absent — and the institution never learns it mattered. The repertory grid is a corrective specifically for this blind spot.

Practical uses include:

• Instrument development, done right. Before locking a standard questionnaire for a programme, run repertory grids with a sample of students to discover which constructs they actually use. The elicited constructs become candidate items — grounding the survey in students'' meaning rather than administrators'' assumptions. This is a more rigorous front end than guessing, and complements cognitive interviewing (which tests whether students understand items you have already written).
• Diagnosing a puzzling result. When a course scores oddly on the standard form, grids with a few students can reveal the hidden dimension driving it — a construct like "whether the teacher seemed to care whether we learned" that no item captured.
• Comparative evaluation. Because elements can be several courses or several sessions, grids reveal how students discriminate between them and on which of their own criteria — richer than a course scoring 4.1 versus 3.8 on imposed items.
• Tracking conceptual development. Repeating grids across a programme can show how students'' criteria for judging teaching and their own work mature — evidence of a kind that programme-level QA and accreditation narratives value but rarely have.

Limitations and honest caveats

The repertory grid is powerful but not a panacea, and a rigorous reader should note real constraints.

First, it is labour-intensive. Traditional elicitation is a one-to-one process; running enough grids to generalise is costly. This has historically confined the technique to small studies and instrument-development work rather than routine census evaluation of every course.

Second, elicitation quality depends heavily on the interviewer. Triadic elicitation can produce shallow or leading constructs if poorly done, and the "right" number of elements and triads involves judgement. Jankowicz (2004) is explicit that the method is a craft.

Third, analysis choices carry assumptions. Grid data can be analysed many ways (cluster analysis, principal components), and treating the elicited rating scales as interval-level involves the same ordinal-versus-interval caution that applies to Likert data generally. Cross-respondent aggregation is genuinely tricky because each person''s constructs differ — the very feature that makes the method valuable also makes pooling delicate.

Fourth, idiosyncrasy cuts both ways. Personal constructs are, by design, personal; some will be unique to one student and not actionable at course level. The method surfaces candidate dimensions that then need judgement about which are shared and consequential.

Fifth, it is a complement, not a replacement. For tracking trends, benchmarking, and accreditation evidence at scale, standard instruments with known psychometric properties remain necessary. The grid''s role is to make sure those instruments are asking about the right things and to illuminate results they cannot explain.

How Koji incorporates this

The historical barrier to repertory grids in routine evaluation has been that structured, one-to-one elicitation does not scale. Koji''s AI-moderated conversational interview is precisely the capability that lowers that barrier, so the spirit of the technique — let students name their own criteria — can operate at a scale traditional grids never reached.

• Eliciting constructs, not just scoring supplied ones. Where a fixed form imposes categories, Koji''s conversational interviews can invite students to articulate, in their own words, what mattered about a course and why — the construct-elicitation move at the heart of the grid. The AI moderator can follow the triadic logic ("you mentioned two courses felt very different — what was the important difference?") to draw out bipolar constructs rather than accepting a bare rating.
• Structured question types to build the matrix. Once a student names a personal construct, Koji''s structured questions (scale, ranking, single_choice) let them rate elements — sessions, activities, or courses — on that construct, approximating the grid''s rating matrix in a self-service, scalable form.
• Automatic thematic analysis to find shared constructs. The hard part of grids at scale is finding which idiosyncratic constructs are actually shared. Koji''s automatic thematic analysis of open-ended responses is designed to cluster students'' self-generated criteria across a cohort, separating the genuinely common dimensions (candidate survey items) from the purely personal ones — directly addressing the aggregation limitation.
• A grounded front end for instrument design. Institutions can use a round of Koji conversational studies to discover the constructs their students actually use before finalising a standard questionnaire, grounding the fixed instrument in student meaning and complementing cognitive-interview pretesting.
• Tracking how criteria evolve. Because Koji can run linked studies over time, it supports the longitudinal use of the technique — observing how students'' constructs for judging teaching and their own work develop across a programme.

The honest framing: Koji does not run a textbook repertory grid with formal triadic elicitation and principal-components analysis, and it should not claim to. What it is designed to do is operationalise the technique''s core insight — that valid evaluation must capture the respondent''s own constructs, not only the designer''s — through conversational elicitation and thematic clustering at a scale that makes it usable for real quality assurance. Koji''s core research platform at koji.so applies the same construct-elicitation engine to product and customer research, where discovering the criteria customers actually use to judge a product — rather than the ones a team assumed — is the difference between a survey that confirms priors and one that learns something.

Related resources

• Q-methodology for student viewpoints
• The critical incident technique for course feedback
• Designing specific open-ended evaluation prompts
• Cognitive interviewing to pretest evaluation questions
• The most significant change technique
• Framework analysis of open-text feedback

References

• Kelly, G. A. (1955). The Psychology of Personal Constructs. New York: Norton.
• Fransella, F., Bell, R., & Bannister, D. (2004). A Manual for Repertory Grid Technique (2nd ed.). Chichester: John Wiley & Sons. https://doi.org/10.1002/0470013370
• Jankowicz, D. (2004). The Easy Guide to Repertory Grids. Chichester: John Wiley & Sons. (ISBN 9780470854044)
• Wright, T. (2003). Exploring the usefulness of Kelly''s personal construct theory in assessing student learning in science courses. Teaching in Higher Education, 8(3), 341–353. https://doi.org/10.1080/13562510309394