Q-Methodology: Surfacing the Distinct Viewpoints Students Hold About a Course

BLUF: Q-methodology is a qualiquantological research technique — invented by William Stephenson in 1935 — that has students rank-order a set of statements about a course onto a forced-choice grid, then applies by-person factor analysis to those rankings. Instead of averaging how students rate a course, it groups students who sorted the statements in similar ways, revealing the two-to-four genuinely distinct "viewpoints" that coexist in a single class. Where a Likert mean tells you the class scored the course 4.1/5, Q tells you which shared stories produced that number — the enthusiasts, the overwhelmed, and the pragmatists — and what each group actually cares about.

What the research says

Q-methodology inverts the logic most quantitative researchers take for granted. In conventional ("R") factor analysis you correlate variables (test items, survey questions) across a large sample of people to find latent traits. Q correlates people across a set of items to find latent viewpoints. William Stephenson — a student of Charles Spearman, the founder of factor analysis — proposed this inversion in a short 1935 letter to Nature titled "Technique of factor analysis," arguing that by transposing the data matrix one could derive factors describing groups of individuals who share a subjective standpoint, rather than groups of correlated variables (Stephenson, 1935). The definitive modern treatment is Watts and Stenner's Doing Q Methodological Research (2012) and their earlier tutorial paper (Watts & Stenner, 2005); Brown's (1993) "A primer on Q methodology" remains the standard reference for the analytic machinery.

The method proceeds through a recognisable sequence of stages.

The concourse. Every Q study begins with the concourse — the full universe of things that can be said about the topic. For a course, the concourse is the entire flow of opinion and commentary surrounding it: what students say in corridors, on forums, in open-text survey boxes, in focus groups. Concourse theory holds that subjectivity is communicable and shareable; the concourse is where it lives (Watts & Stenner, 2012).

The Q-set. From this concourse the researcher samples a manageable, balanced set of statements — typically 40 to 60 — that represents its breadth. These become the Q-set: cards each bearing one statement, e.g. "The assessment felt disconnected from what we were taught," or "I always knew what was expected of me." Statement-set design is where craft matters most; the set must span the concourse without loading it toward any one position.

The P-set. The P-set is the group of participants who will perform the sort. Crucially, Q is small-N by design. A well-structured Q study often uses 20–40 participants, sometimes fewer, because the unit of analysis is the viewpoint, not the individual — and you need only enough people to define and saturate each viewpoint several times over, not a representative population sample (Brown, 1993).

The Q-sort. Each participant rank-orders the whole Q-set onto a quasi-normal, forced-choice grid running from, say, −5 ("least like how I feel") through 0 to +5 ("most like how I feel"). The grid has fixed slot counts — many neutral slots in the middle, few at the extremes — so participants must make trade-offs, placing only a couple of statements at each pole. This forced distribution is not a statistical assumption about the data; it is a device that compels participants to model their own point of view holistically, weighing every statement against every other (Watts & Stenner, 2005).

By-person factor analysis. The completed sorts — each a column of scores — are inter-correlated person against person and factor-analysed. Participants whose sorts correlate highly load onto the same factor. Each extracted factor is a mathematically-derived shared viewpoint: an idealised sort that represents how a hypothetical person holding that viewpoint would have arranged the statements. Factors are typically rotated (varimax or by-hand/theoretical rotation) and then interpreted.

Interpreting factors as viewpoints. Interpretation is the qualitative payoff. The researcher reconstructs each factor's crib sheet — the statements it ranks highest and lowest, and the distinguishing statements that separate it from the other factors — and writes a rich narrative account of the standpoint it embodies. This is why Watts and Stenner call Q qualiquantological: the factor extraction is genuinely quantitative and replicable, but the product is an interpretive, qualitatively-textured portrait of a shared subjectivity.

The method has a real, if modest, track record in higher education. Zaitseva and Law (2022) used Q-methodology to identify student priorities in module-level experience and found three distinctive viewpoints corresponding to different stages of the student journey, levels of maturity, and degrees of cognitive engagement — a structure entirely invisible to a mean satisfaction score. Their study is a clean demonstration of the core claim: a single module does not produce one student experience but several coherent ones, and Q surfaces them as nameable, actionable groups.

Why it matters for course evaluation in practice

The dominant instrument in course evaluation — the Likert-scale Student Evaluation of Teaching (SET) — reports a mean and perhaps a standard deviation per item. The mean is a powerful compression, and that is exactly the problem: it assumes the class is a single population drawn around one central tendency. Real cohorts rarely are.

Consider a large first-year methods course that scores 4.0/5 on "overall satisfaction." That number is consistent with wildly different underlying structures. It could be a homogeneous class of mildly-satisfied students. Or — far more commonly — it could be the average of three distinct groups: a confident quantitative cohort who found the course too slow (rating it 3), a group who found the pacing perfect (rating it 5), and an anxious group who were drowning but appreciated the instructor's warmth (rating it 4). The mean of 4.0 describes none of these three real students. Worse, the standard deviation that flags the disagreement gives you no purchase on what the disagreement is about — only that it exists.

Q-methodology is built precisely to recover that hidden structure. Because it groups people by the shape of their whole response pattern rather than by their score on any one item, it separates the "too slow" viewpoint from the "drowning" viewpoint even when both produce middling overall ratings for entirely different reasons. The output is not a number to defend at a committee meeting but a set of two-to-four personas — each with a name, a narrative, and a list of the statements that define and distinguish it. A programme director can act on "the 30% of students who experience the assessment as disconnected from teaching" in a way they simply cannot act on "SD = 1.1." Q turns dispersion from noise into signal.

This is the same reason Q complements — rather than replaces — SET averages and multidimensional instruments like the SEEQ. Averages answer how much; Q answers for whom, and in what pattern. Used together, a department can track headline satisfaction over time while periodically running a Q study to understand the qualitatively distinct experiences that the headline number conceals.

Limitations and honest caveats

Q-methodology is not a free lunch, and a critical academic will raise several objections that deserve honest answers.

It does not tell you prevalence. This is the single most important caveat. Q identifies which viewpoints exist and what they consist of; it does not validly estimate what proportion of the population holds each one. The number of participants loading on a factor is an artefact of who you recruited, not a population estimate. Reporting "40% of students hold Viewpoint A" from a Q study is a category error. If you need prevalence, you follow the Q study with a large-sample survey that operationalises the viewpoints Q discovered — Q generates the hypotheses; R-method tests their distribution.

Administration burden. A Q-sort is cognitively and logistically heavier than ticking a Likert grid. Sorting 40–60 statements onto a forced grid takes 20–40 minutes and ideally a short post-sort interview to capture reasoning. This makes routine, every-course, every-semester deployment impractical; Q is a periodic deep-dive instrument, not a continuous monitor.

Statement-set subjectivity. The Q-set is designed by researchers, and the concourse can never be sampled perfectly. A biased, unbalanced, or poorly-worded statement set constrains what viewpoints can possibly emerge — you cannot discover a standpoint your statements gave participants no way to express. Rigor here depends on transparent, defensible concourse sampling, which is labour-intensive and never fully escapes the designer's framing.

Small P-set and factor stability. Because N is small by design, decisions about how many factors to extract and retain, and how to rotate them, involve genuine judgement. Different defensible analytic choices can yield somewhat different factor solutions. Watts and Stenner (2012) are candid that Q is not a mechanical procedure; two competent analysts may interpret the same factor array with different emphases.

Interpretation is judgement. The final narratives are written by a human reading crib sheets and distinguishing statements. This is a strength (it produces meaning) and a vulnerability (it is not blind to the analyst's priors). Best practice — member-checking factor interpretations with participants, documenting the analytic audit trail — mitigates but does not eliminate this.

In short: Q buys you depth and structure at the cost of generalisability and effort. It answers a question SET averages cannot, but it does not answer the questions SET averages are good at.

How Koji incorporates this

Koji for Education is not a Q factor-analysis engine, and we will not claim it is. But Koji is designed around the same underlying conviction that animates Q-methodology: a class contains several distinct, coherent student experiences, and the job of evaluation is to surface them rather than dissolve them into a mean.

Several Koji mechanisms map onto Q's logic in spirit and in function:

• AI-moderated conversational interviews replace the static survey box with a moderated dialogue. When a student gives an open-text answer, Koji probes it with adaptive follow-up questions, drawing out the narrative behind the rating — much as a Q post-sort interview draws out the reasoning behind a sort. This is how Koji builds up the raw material of the concourse: authentic, first-person student commentary rather than pre-boxed options.

• Automatic thematic analysis that clusters open text into recurring viewpoints is Koji's structural analogue to by-person factoring. Rather than reporting a single average, Koji's reporting layer groups responses into recurring themes and reader-facing segments — surfacing the two-to-four distinct stories in a cohort as named clusters with supporting quotations, co-occurring themes, and sentiment, instead of collapsing them into one number.

• Structured question types give evaluators a spectrum of instruments in one study: open_ended (free-form, with AI follow-up probing), scale, single_choice, multiple_choice, and ranking. The ranking type in particular mirrors the Q-sort's forced-choice logic — it makes students order items by preference and reports average position, compelling the same kind of trade-off that a Q grid enforces, rather than letting everything score 4/5.

• Bias-aware, segment-aware reporting keeps distinct viewpoints legible in the output. Findings are attributed to segments and traced back to citable source quotes, so a dean reads "the group who experienced the assessment as disconnected" as a distinct account — not as variance around a mean.

• Triangulation across cohorts lets institutions compare how these viewpoint-structures shift between sections, semesters, or delivery modes, giving the longitudinal read that a one-off Q study cannot.

The honest framing: Koji is engineered to surface distinct viewpoints — the outcome Q is prized for — through conversational depth and thematic clustering, not by literally running Stephenson's by-person factor analysis. For institutions that want the full methodological rigor of Q for a high-stakes curriculum review, Koji's outputs make an excellent concourse-building and hypothesis-generating front end to a formal Q study.

This viewpoint-surfacing engine is not education-specific. Koji's core platform at koji.so applies the same conversational-interview-plus-thematic-clustering approach to product and customer segmentation research — the same problem of finding the distinct segments hidden inside an aggregate, in a different domain.

References

• Brown, S. R. (1993). A primer on Q methodology. Operant Subjectivity, 16(3/4), 91–138. https://ojs.library.okstate.edu/osu/index.php/osub/article/view/9020
• Stephenson, W. (1935). Technique of factor analysis. Nature, 136(3434), 297. https://doi.org/10.1038/136297b0
• Watts, S., & Stenner, P. (2005). Doing Q methodology: theory, method and interpretation. Qualitative Research in Psychology, 2(1), 67–91. https://doi.org/10.1191/1478088705qp022oa
• Watts, S., & Stenner, P. (2012). Doing Q Methodological Research: Theory, Method and Interpretation. London: SAGE. ISBN 9781849204156.
• Zaitseva, E., & Law, A. (2022). Questions that matter: using Q methodology to identify student priorities in module level experience. Quality in Higher Education, 29(2), 261–278. https://doi.org/10.1080/13538322.2022.2100101

Related Resources

• What Can Open-Text Student Comments Tell You That Likert Scores Cannot?
• What Do Student Evaluations Actually Measure? Marsh, the SEEQ, and Multidimensionality
• Adaptive Comparative Judgement in Evaluation
• Best-Worst Scaling (MaxDiff) for Course-Evaluation Priorities
• The Framework Method for Open-Text Course Feedback
• Illuminative Evaluation: Parlett and Hamilton's Learning Milieu