Network Psychometrics: Treating Course-Evaluation Items as a Network, Not a Hidden Factor

In brief

Network psychometrics is an approach that models a set of course-evaluation items as a network of directly interacting responses rather than as noisy reflections of one or more hidden latent factors. Each item is a node; each edge is a partial correlation — the association between two items after conditioning on every other item. The payoff is a different question: instead of "how many dimensions underlie the survey?", you ask "which items are central, which items bridge otherwise separate clusters, and which associations survive once everything else is held constant?" The methodology is well developed and comes with rigorous stability diagnostics, but with the small samples of a single course the estimated network is fragile, and centrality metrics must be read with real caution.

What the research says

The network approach was introduced to psychology as an alternative to the common-cause (reflective latent-variable) model. Borsboom and Cramer (2013), in the Annual Review of Clinical Psychology, argued that many constructs are better understood not as a single underlying entity that causes its indicators, but as systems in which the observed variables directly influence one another. In that framing a "construct" is an emergent property of a densely connected cluster, not a hidden cause.

The estimation machinery was made practical and reproducible by Epskamp, Borsboom and Fried (2018) in Behavior Research Methods. Their tutorial establishes the now-standard workflow: estimate a Gaussian graphical model (a partial-correlation network), regularise it — typically with the graphical LASSO tuned by an information criterion — so that small, likely-spurious edges are shrunk to exactly zero, and then, crucially, quantify how stable the results are. They introduce the correlation-stability (CS) coefficient and bootstrapped confidence intervals for edges, and warn that centrality indices — especially betweenness and closeness — are frequently unstable in samples of the size psychologists actually collect. Fried and Cramer (2017), in Perspectives on Psychological Science, catalogue the methodological challenges candidly: replicability, the interpretation of centrality, and the risk of reifying a data-driven graph.

Translated to course evaluation, a network model of a teaching-quality instrument might show that "the assessment reflected the course content" and "feedback helped me improve" are directly connected, that a global "I would recommend this course" item sits centrally, and that "workload was manageable" bridges an "organisation" cluster and a "wellbeing" cluster — structure that a single overall mean, or even a factor score, would flatten.

Why it matters for course evaluation in practice

Course-evaluation instruments are almost always analysed as if a latent "teaching quality" causes every response — an assumption underlying most work on dimensionality and what student ratings measure. The network lens offers a complementary reading. A central item is one that is tightly wired to the rest of the questionnaire; if you could only ask three questions, high-centrality items are strong candidates. A bridge item connects clusters that would otherwise be independent — often the most actionable finding, because intervening on a bridge (say, clarifying assessment expectations) may propagate to items in both clusters.

This makes network analysis a natural companion to prioritisation tools such as importance-performance analysis and the Kano model: where those rank items by importance and performance, a network shows the connectivity that explains why moving one item might move others. It also gives a principled counterweight to common-method bias worries — a densely connected core partly reflects shared method variance, and the regularised network makes that dependence visible rather than hidden inside a single scale score.

Limitations and honest caveats

The critical reader should treat a single-course network with suspicion, for good reasons.

Sample size and stability. Regularised network estimation needs substantial samples; the number of parameters grows quadratically with the number of items. Epskamp et al.'s own diagnostics routinely show that centrality orderings — particularly betweenness and closeness — are not stable below samples in the many hundreds. A network drawn from thirty seminar responses is close to noise. Report the CS coefficient and bootstrapped edge intervals, or do not interpret the graph.

Correlational, cross-sectional, undirected. A standard Gaussian graphical model gives undirected edges from one wave of data. It cannot tell you whether clear assessment drives perceived fairness or the reverse. Reading causal or intervention implications off an undirected cross-sectional network is exactly the over-claim Fried and Cramer warn against.

Regularisation choices shape the picture. The LASSO tuning parameter decides which edges survive; a more conservative penalty yields a sparser, different-looking network. As with any analysis with researcher degrees of freedom, the graph is one defensible result among several.

Centrality is not importance-for-action. A central item is statistically well connected; that is not the same as being the lever a programme director should pull. Betweenness in particular is both unstable and easily misread.

Networks versus factors is partly philosophical. For many instruments a latent-variable model and a network model fit comparably; the choice is as much about the question you want to ask as about which is "true."

How Koji incorporates this

A network is only as good as the item set feeding it, and this is where Koji's design helps. Because Koji fields structured questions across several types (open_ended, scale, single_choice, multiple_choice, ranking, yes_no), it produces the rich, multivariate response matrix that a partial-correlation network needs — rather than the single global rating that collapses all structure. Its automatic thematic analysis of open-text feedback can generate additional coded indicators, so that themes surfaced in students' own words can enter the network alongside the scale items and reveal which qualitative concerns bridge otherwise separate numeric clusters.

Koji's aggregate, programme-level reporting is the right altitude for this method: by pooling responses across many sections and cohorts, it reaches the sample sizes at which a regularised network and its stability diagnostics become trustworthy — the single-seminar network is precisely what the evidence says not to trust. And because Koji supports repeated, mid-cycle collection, the platform can move beyond a single cross-sectional snapshot toward the kind of multi-wave data that more advanced (temporal) network models require to say anything about direction.

We are careful to frame this as designed to support network-style structural analysis at scale, not as a claim that Koji delivers a validated causal network for every course. Koji's core research platform at koji.so applies the same AI-moderated interview engine to product and customer research, where mapping which attributes of an experience reinforce one another is an equally central analytical goal.

Related resources

• What Do Student Evaluations Actually Measure? Marsh, the SEEQ, and Multidimensional Feedback
• One Number or Many? The Dimensionality Debate
• How Many Dimensions Does Your Evaluation Really Measure? Parallel Analysis
• Importance-Performance Analysis: Turning Scores into a Priority Map
• Why a Single Student Survey Cannot Stand Alone: Common-Method Bias

Frequently asked questions

What is the difference between a network model and a factor model of a survey?

A factor model assumes a hidden latent variable causes the item responses, so correlations between items are explained by their shared dependence on that factor. A network model makes no such assumption: it estimates direct associations (partial correlations) between items after conditioning on all others, and treats a construct as an emergent property of a densely connected cluster rather than a hidden cause.

What does it mean for a course-evaluation item to be central or a bridge?

A central item is tightly connected to many others in the network, so it is a strong candidate if you must shorten the survey. A bridge item links two otherwise-separate clusters — for example connecting an organisation cluster to a wellbeing cluster — and is often the most actionable, because change there may propagate into both clusters.

Can I run a network on one class of thirty students?

Not reliably. Regularised network estimation has many parameters and needs substantial samples; centrality orderings are typically unstable below several hundred respondents. A thirty-response network is close to noise, so network analysis belongs at programme or cohort level where sections are pooled.

Does a network tell me what causes high evaluations?

No. A standard Gaussian graphical model produces undirected edges from cross-sectional data, so it cannot establish direction or causation. Reading intervention implications off an undirected, one-wave network is a common over-claim; temporal network models on multi-wave data are needed even to begin addressing direction.

How do I know the network is trustworthy and not an artefact?

Report stability diagnostics: the correlation-stability (CS) coefficient for centrality and bootstrapped confidence intervals for the edges. If edges have wide overlapping intervals or the CS coefficient is low, the ordering of items by centrality should not be interpreted.

Is the network approach better than factor analysis for course evaluation?

Neither is universally better. For many instruments the two fit comparably, and the choice depends on the question: factor analysis is suited to scoring an underlying dimension, while a network is suited to seeing which items reinforce one another and which bridge clusters. They are complementary readings of the same data.

References

• Borsboom, D., & Cramer, A. O. J. (2013). Network analysis: An integrative approach to the structure of psychopathology. Annual Review of Clinical Psychology, 9, 91-121. https://doi.org/10.1146/annurev-clinpsy-050212-185608
• Epskamp, S., Borsboom, D., & Fried, E. I. (2018). Estimating psychological networks and their accuracy: A tutorial paper. Behavior Research Methods, 50(1), 195-212. https://doi.org/10.3758/s13428-017-0862-1
• Fried, E. I., & Cramer, A. O. J. (2017). Moving forward: Challenges and directions for psychopathological network theory and methodology. Perspectives on Psychological Science, 12(6), 999-1020. https://doi.org/10.1177/1745691617705892