The Blind Spot in Course Evaluation: Your Survey Never Checks What the Job Market Is Asking For

The short version

A course evaluation is an inward-looking instrument. It asks students what they experienced, how satisfied they were, sometimes what they think they learned. What it almost never does is check any of that against a single external fact: what the labour market your graduates are walking into is actually asking for. That gap matters, because the evidence says the market and the curriculum drift apart. Around a quarter of highly qualified young employees in the EU are working in jobs below their qualification level, and recent graduates are roughly twice as likely to be overqualified in their first job as the cohort of the 1990s (Cedefop European Skills and Jobs Survey). A programme can score 4.3 on "the course met my expectations" while quietly failing the external test that decides whether those expectations were the right ones.

Real-time labour-market intelligence now makes that external test cheap to run. The question is whether quality assurance will use it, and use it without letting it hollow out the curriculum.

What "labour-market intelligence" now means

For most of the history of programme review, the labour market was something you consulted every few years through an advisory board, an alumni survey, or an accreditation panel. The data was slow, small, and retrospective. That has changed.

Tools like Cedefop's Skills-OVATE (the Skills Online Vacancy Analysis Tool for Europe) now mine online job advertisements across the EU27 and EFTA countries, extracting the occupations, tasks and specific skills employers are advertising for, updated quarterly and close to real time (Cedefop Skills-OVATE). Built on a memorandum of understanding with the European Labour Authority and Eurostat's Web Intelligence Hub, it applies natural-language processing to millions of postings to produce a live map of demand by country, region, sector and occupation. Commercial equivalents such as Lightcast do the same at finer grain.

For a programme director, this is a new kind of mirror. Instead of asking "were students satisfied?", you can ask "did the skills students report developing match the skills employers in this region are advertising for, this quarter, not in the last accreditation cycle?"

Why course evaluation needs it

The case is not that satisfaction data is worthless; it is that satisfaction data has no external referent. We have argued before that course evaluation cannot directly measure employability and that graduate-outcome signals arrive too late to fix a course. Labour-market intelligence sits neatly between those two problems. It is not a lagging destination metric (which takes years to arrive), and it is not an internal perception (which has no anchor). It is a leading external signal available now.

Used well, it does three things a student survey cannot:

1. It validates the construct. If your evaluation asks students whether they developed "data-handling skills," vacancy analysis can tell you whether that phrase maps onto what employers in your graduates' target sector actually request, or whether you are teaching a skill nobody is hiring for under that name.
2. It prioritises. The Cedefop survey found that around 39% of adult EU employees are overskilled and stuck in low-quality jobs, while over half of those who found work after 2011 reported few opportunities to land a role matched to their qualifications. Demand data helps a programme distinguish a genuine skills gap from a mismatch of expectations.
3. It closes the employer loop cheaply. Advisory boards give you a dozen opinions once a year; vacancy analysis gives you an aggregate of hundreds of thousands of hiring decisions continuously, a complement to the employer feedback loop most programmes run far too rarely.

"But isn't this just teaching to the job market?"

Here is the objection a serious academic audience will raise, and it is the right one.

Job ads are a biased sample of the economy. Online vacancy data over-represents digitally advertised, higher-skill, private-sector and urban roles, and under-represents public-sector hiring, SMEs, word-of-mouth recruitment and whole regions. Cedefop is explicit that OVATE complements, and does not replace, established labour-market intelligence. Treat a skills-demand chart as the whole truth and you will systematically under-weight the parts of the economy that do not advertise online.

Demand is not the same as educational purpose. A university is not a job-training pipeline. Chasing this quarter's advertised skills risks narrowing a degree to a bundle of short-lived tool competencies, at the expense of the durable capacities, critical reasoning, adaptability, disciplinary depth, that outlast any specific vacancy. The employability research is clear that these deeper capacities matter; see our piece on the graduate capital model. Labour-market data should inform the conversation, not dictate the curriculum.

Signals lag reality too. By the time a skill floods the job ads, the market may already be saturating. Demand data is a compass, not a crystal ball.

The honest position, then, is narrow: labour-market intelligence is a powerful external validity check on what a programme claims to develop, one triangulation source among several, not a mandate. It answers "is our stated skill set still connected to the world our graduates enter?" It does not answer "what should a degree be for?" That remains an academic judgement.

What this looks like in practice

Making this concrete matters, because "consult labour-market data" can collapse into a slide nobody acts on. A workable loop has four steps. First, at the end of a programme, ask students to rank the capabilities they actually developed, not rate each one in isolation; ranking forces the trade-offs a five-point scale hides. Second, pull the current demand profile for the occupations your graduates target from Skills-OVATE or a commercial equivalent, and reduce it to the ten or fifteen skills that dominate the vacancies. Third, lay the two lists side by side and look for the gaps that matter: a capability high in demand but low in what students report building is a curriculum signal; a capability students rank highly but that barely appears in vacancies is worth a conversation about whether you are teaching for a market that has moved. Fourth, feed the comparison into the periodic review the ESG expect anyway, so it becomes evidence rather than an interesting aside.

Two cautions keep the exercise honest. The demand profile is a snapshot of advertised roles, so weight it against what you know about the parts of your graduates' destinations that hire off-platform. And pair it with actual destination data where you have it: graduate tracer studies tell you where graduates landed, which is the ground truth that vacancy data only approximates. Demand signals and destination signals disagree often enough that holding both is the whole point.

Where Koji fits

Labour-market intelligence and course evaluation are two halves of a loop that most institutions never close. OVATE tells you what the market demands. Your evaluation tells you what students experienced. Nobody joins them.

Koji for Education is built to join them. Rather than a static Likert form, its AI-moderated conversational interviews can ask students, at programme level, which capabilities they actually built and where they felt under-prepared, and probe those answers instead of averaging a number. Its six structured question types (open_ended, scale, single_choice, multiple_choice, ranking, yes_no) let you have students rank the skills they developed, which you can then hold up against a live demand profile for their sector. Its automatic thematic analysis surfaces the capabilities students mention unprompted, an internal signal you can compare with external demand. And its closing-the-loop action tracking and programme- and institution-level reporting turn that comparison into a documented change, exactly the kind of evidence accreditors and ESG-aligned quality processes reward. Because Koji is GDPR-compliant and EU-appropriate by design, none of this requires exporting student data into ad-tech pipelines.

The same conversational interview engine underpins the general-purpose Koji platform, so a careers service or employer-relations team running its own stakeholder interviews can work from the same evidence base.

The takeaway

Course evaluation without labour-market intelligence is a mirror with no window: it shows the classroom in fine detail and the outside world not at all. Adding a live demand signal gives your evaluation an external referent for the first time. Just remember what kind of instrument it is: a compass for keeping a programme honest about its claims, not a curriculum committee that has quietly outsourced the purpose of education to whoever is hiring this quarter.