How Many Dimensions Does Your Evaluation Really Measure? Parallel Analysis for Factor Retention

In brief

Before you can report an evaluation instrument as measuring "clarity", "workload", and "assessment" as separate dimensions, you have to decide how many dimensions the items actually support — a factor-retention decision. The two methods most institutions default to (the eigenvalue-greater-than-one rule and eyeballing a scree plot) are demonstrably poor and tend to over-extract factors. Parallel analysis (Horn, 1965) fixes this by comparing your data's eigenvalues against those from random data of the same size, retaining only factors that beat chance. Across simulation studies it is one of the most accurate retention methods available (Zwick & Velicer, 1986) — and one of the most underused (Hayton, Allen & Scarpello, 2004).

What the research says

When you factor-analyse a set of evaluation items, you must choose how many factors to retain. For decades the default was Henry Kaiser's rule: keep every factor with an eigenvalue greater than 1.0 (the "K1" rule). John Horn showed in a 1965 Psychometrika paper why this is wrong. Even in random data with no underlying structure, sampling error and least-squares capitalisation push some eigenvalues above 1.0. The K1 rule counts that noise as signal and systematically over-extracts. Horn's solution — parallel analysis — is elegant: generate many datasets of random numbers with the same number of variables and observations as your real data, compute their eigenvalues, and retain a real factor only if its eigenvalue exceeds what random data of that size produce (typically the 95th percentile of the random eigenvalue distribution). You keep only the factors that beat noise.

The evidence that this matters is strong. Zwick and Velicer (1986), in a much-cited Psychological Bulletin simulation, compared five retention rules — Kaiser's K1, Cattell's scree test, Bartlett's chi-square test, Velicer's minimum average partial (MAP), and parallel analysis — across systematically varied conditions. Parallel analysis and MAP performed best, while the K1 rule was among the least accurate, frequently over-estimating the number of components. The scree test, though better than K1, is subjective and unreliable between analysts.

Despite this, adoption lagged. Hayton, Allen and Scarpello (2004) documented in Organizational Research Methods that parallel analysis remained "one of the most underutilised" retention methods in applied research even though it is one of the most accurate, and they provided a step-by-step tutorial to close the gap. Parallel analysis is now available in standard statistical software (R's psych::fa.parallel, dedicated SPSS/SAS macros, jamovi, and JASP), so the practical barrier is largely gone.

Why it matters for course evaluation in practice

Factor-retention decisions are not an academic nicety for course evaluation — they determine what you are allowed to report and how.

Suppose your evaluation form has 15 items and you want to report sub-scores for distinct teaching qualities. The number of defensible sub-scores is exactly the number of factors the data support. If you use the K1 rule and it tells you there are five factors when parallel analysis says there are two, you will invent three "dimensions" that are really noise — reporting an "assessment" sub-score and a "feedback" sub-score as separate signals when the items actually measure one thing. Worse, those spurious sub-scores may then feed into personnel or programme decisions, giving unfair, unreliable numbers the appearance of precision.

The dimensionality question sits directly upstream of several things your institution already does:

• Reliability reporting. Cronbach's alpha and coefficient omega assume you know which items belong to which scale. Get the factor structure wrong and the reliability coefficients describe a scale that does not exist.
• The global-vs-profile debate. Whether student ratings are best used as one overall number or as a multidimensional profile (a long-running question in the SET literature) is, at bottom, a factor-retention question. Parallel analysis gives it an empirical, replicable answer for your instrument rather than importing someone else's.
• Instrument revision. When you add or cut items, the factor structure can change. Re-running parallel analysis tells you whether your revised form still supports the sub-scores you report.

The core message is a discipline: do not report a sub-score you cannot justify as a retained factor, and do not decide the number of factors with a rule (K1) that the methodological literature has known to be poor since 1986.

Limitations and honest caveats

Parallel analysis is a strong default, not an oracle, and a careful psychometrician should know its boundaries.

• It is a guide, not a verdict. Retention should triangulate parallel analysis and MAP with theory and interpretability. A statistically "retained" factor loaded by two barely-related items may be less useful than a theoretically coherent one that just misses the cutoff. Numbers inform judgement; they do not replace it.
• Method variants disagree. Parallel analysis on principal-components eigenvalues (the original) can retain a different number than parallel analysis on common-factor eigenvalues, and using the mean versus the 95th-percentile random eigenvalue changes results. Report which variant you ran; the differences are not trivial.
• It can under-extract with correlated factors. When real factors are themselves strongly correlated, parallel analysis can occasionally miss a genuine factor. It is more accurate than K1 but not immune to error.
• Ordinal, skewed items need the right correlation matrix. Course ratings are coarse ordinal scales. Running parallel analysis on Pearson correlations can distort results; polychoric correlations are generally more appropriate, and the random-data generation should respect the data's discreteness.
• Sample size still governs stability. Factor analysis of a single small class is unstable regardless of the retention rule. Parallel analysis does not rescue an underpowered dataset — it needs an adequate respondent-to-item ratio to give trustworthy answers.

Used as the primary retention criterion, cross-checked with MAP and interpretability and reported transparently, it is the responsible choice; treated as an automatic decision rule, it can still mislead.

How Koji incorporates this

Koji's stance is that instrument structure should be earned from evidence, and the platform is built to make that discipline routine rather than exceptional:

• Structure that reflects the data, not habit. Koji encourages reporting only the dimensions an instrument genuinely supports, so evaluation reports do not proliferate spurious sub-scores. The factor-retention question — how many real dimensions are here — is treated as a prerequisite to sub-score reporting, not an afterthought.
• Beyond forcing items into pre-set scales. Because Koji's AI-moderated conversational interviews and open-ended items surface what students actually raise, dimensions can emerge from evidence rather than being imposed by a form designer's assumptions — a qualitative complement to the quantitative retention decision, and a hedge against a rigid factor structure that the data do not support.
• Reliability that follows structure. Koji's reporting ties reliability evidence to the scale structure, reinforcing that a Cronbach's alpha or omega is only meaningful once the dimensionality question has been answered honestly.
• Adequate data for stable analysis. By supporting higher-quality response collection across cohorts, Koji helps teams reach the respondent-to-item ratios that make any factor-retention method — parallel analysis included — trustworthy.

We frame this as supporting evidence-based instrument design, not automating psychometric judgement: deciding the number of factors and which sub-scores to report remains a human, methodological act that parallel analysis informs. Teams building and validating survey instruments for product and customer research can apply the same retention discipline through Koji's core platform at koji.so.

Frequently asked questions

What is parallel analysis in one sentence? It is a factor-retention method (Horn, 1965) that compares your data's eigenvalues against eigenvalues from random data of the same size, retaining only the factors whose eigenvalues exceed what random noise produces.

Why not just use the eigenvalue-greater-than-one rule? Because even random data produce eigenvalues above 1.0 through sampling error, so the K1 rule counts noise as signal and systematically over-extracts factors. Simulation studies (Zwick & Velicer, 1986) rank it among the least accurate retention methods.

Isn't the scree plot good enough? The scree test is better than K1 but subjective — different analysts read the "elbow" differently, so it is not reproducible. Parallel analysis gives a defensible, replicable cutoff and outperforms scree in simulations.

How does this affect the sub-scores we report? Directly. The number of defensible sub-scores equals the number of retained factors. If parallel analysis supports two factors but you report five sub-scores, three of them are noise dressed up as distinct teaching qualities — and may feed unfairly into decisions.

Does parallel analysis work on 1-5 Likert items? With adaptation. Ratings are coarse and ordinal, so parallel analysis on ordinary Pearson correlations can distort results; use polychoric correlations and generate the random data to respect the items' discreteness.

Should parallel analysis be the only thing we look at? No. Triangulate it with Velicer's MAP, theoretical coherence, and factor interpretability. A retained factor that makes no substantive sense, or a near-miss factor that is theoretically clear, both deserve judgement beyond the cutoff.

Related resources

• One Number or Many? The Dimensionality Debate and Personnel Decisions
• What Do Student Evaluations Actually Measure? Marsh, the SEEQ, and Multidimensional Feedback
• Coefficient Omega vs Cronbach's Alpha: Reporting Scale Reliability
• Cronbach's Alpha and Course-Evaluation Reliability
• Is One "Overall" Question Enough? Single-Item vs Multi-Item Measures
• Measurement Invariance: Can You Compare Scores Across Groups at All?

References

• Horn, J. L. (1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30(2), 179-185. https://doi.org/10.1007/BF02289447
• Zwick, W. R., & Velicer, W. F. (1986). Comparison of five rules for determining the number of components to retain. Psychological Bulletin, 99(3), 432-442. https://doi.org/10.1037/0033-2909.99.3.432
• Hayton, J. C., Allen, D. G., & Scarpello, V. (2004). Factor retention decisions in exploratory factor analysis: A tutorial on parallel analysis. Organizational Research Methods, 7(2), 191-205. https://doi.org/10.1177/1094428104263675
• Velicer, W. F. (1976). Determining the number of components from the matrix of partial correlations. Psychometrika, 41(3), 321-327. https://doi.org/10.1007/BF02293557