AI governance

EU AI Act classification

The EU AI Act (Regulation 2024/1689) establishes a risk-based framework for artificial intelligence systems deployed in the European Union. Annex III, Section 3 of the regulation identifies AI systems intended for use in education and vocational training as potentially high-risk. This includes systems that may determine access to education, evaluate learning outcomes, or assess the level of education an individual will receive.

Koji has completed a formal classification assessment in accordance with Article 6(3) of the regulation. Our assessment considered the specific functions the platform performs in the context of higher education quality assurance.

Assessment outcome

The Koji platform does not determine access to education, evaluate individual learning outcomes, or assess the level of education a student will receive. Its primary function is to collect and analyse student experience feedback through AI-conducted interviews, producing quality scores and thematic summaries for institutional review.

However, the outputs Koji generates may inform institutional decisions about course delivery, teaching methodology, and resource allocation. In some cases, quality reports may constitute evidence submitted for programme accreditation or periodic review. Because these downstream uses could have a material effect on educational provision, we have elected to treat the system as high-risk and comply with the corresponding obligations under Title III, Chapter 2 of the regulation. This is the conservative, safer classification. We believe it is the responsible approach for a system operating in the education sector, even where an argument for limited-risk classification could be made.

Timeline

Under the current regulation, obligations for high-risk AI systems apply from 2 August 2026. The European Commission has proposed amendments under the Digital Omnibus package that would extend this deadline to 2 December 2027 for certain categories of high-risk systems. Regardless of whether the extension is adopted, Koji is working toward compliance with the original August 2026 deadline.

Risk management

Article 9 of the EU AI Act requires high-risk AI systems to operate under a continuous, lifecycle-spanning risk management system. Koji maintains a living risk register that is reviewed and updated on a quarterly basis.

Identification of foreseeable risks

The following risks have been identified through internal assessment and stakeholder consultation:

• AI hallucination in summaries. Large language models may generate claims that are not supported by the underlying transcript data. This could lead to inaccurate quality reports.
• Quality score bias. Scoring rubrics applied by the AI may systematically disadvantage certain student cohorts, including non-native speakers or students with non-standard communication styles.
• PII leakage. Student data could be inadvertently included in outputs that are shared with users who should not have access to personally identifiable information.

Risk mitigation measures

• Source tracing for all summarised claims, linking every statement in a report back to specific transcript segments.
• Automated PII redaction applied before data is sent to LLM inference, with human review of redaction accuracy.
• Quality scoring rubrics weighted toward substantive content rather than linguistic sophistication.
• Regular bias audits across demographic cohorts to detect systematic disparities in scoring.

Residual risk documentation

Not all risks can be fully eliminated. Residual risks are documented alongside their likelihood and potential impact. These are communicated to deploying institutions so they can factor them into their own risk assessments. We update residual risk documentation whenever mitigations change or new risks are identified.

Post-market monitoring

Koji operates a post-market monitoring programme that tracks output quality, user feedback, and adverse events in production. When issues are detected, they are fed back into the risk management cycle for assessment and mitigation.

Data governance

Article 10 of the regulation sets requirements for data quality and governance in high-risk AI systems.

Training data

Koji does not train its own AI models. Each university connects Koji to their own enterprise LLM accounts, meaning conversation data flows through the institution's existing AI agreements. Koji has no access to, and exercises no control over, the training datasets used by underlying model providers. For institutions without an enterprise LLM agreement, Koji offers EU-native LLM alternatives on request.

Prompt engineering and system instructions

The behaviour of AI components is governed by prompt engineering and system instructions authored by Koji. These instructions are reviewed for potential bias, tested against diverse input scenarios, and versioned in source control. Changes to system prompts undergo internal review before deployment.

Quality scoring rubrics

Quality scoring rubrics define the criteria and weightings used to evaluate student feedback. These rubrics are documented, auditable, and available to deploying institutions on request. Rubric design prioritises substantive content and avoids penalising non-standard language or communication styles.

Transparency

Article 13 requires that high-risk AI systems be designed and developed to ensure their operation is sufficiently transparent to enable deployers to interpret outputs and use them appropriately.

Disclosure to students

Students are clearly informed that they are interacting with an AI system before any conversation begins. The AI interviewer identifies itself as artificial intelligence at the start of every session. There is no attempt to present the AI as human.

Methodology documentation

The quality scoring methodology is documented and made available to the deploying university. This documentation covers the scoring criteria, how rubrics are applied, what weight each dimension carries, and how aggregate scores are calculated.

Traceability of summarisation

Every claim in a Koji quality report links back to the source transcript segments from which it was derived. This allows faculty and quality officers to verify that AI-generated summaries accurately reflect student feedback, and to identify cases where the AI may have misinterpreted or overgeneralised.

Model cards

Model cards describing the AI components used in the platform, including their intended use, known limitations, and performance characteristics, are available on request.

Human oversight

Article 14 requires that high-risk AI systems be designed to allow effective oversight by natural persons. Koji is built around the principle that AI outputs are advisory, not determinative.

Advisory outputs

AI-generated summaries and quality scores are presented as analytical tools to support human decision-making. They are not final determinations. No consequential decisions, such as those relating to faculty promotion, tenure decisions, or programme closure, are made by the AI system.

Access to raw data

Faculty and quality officers can review the raw transcripts alongside AI-generated outputs at any time. This ensures that human reviewers are never limited to the AI's interpretation of the data.

Override capability

Quality scores generated by the AI can be overridden by the institution. The system is designed so that human operators can disregard AI outputs entirely and rely solely on the underlying transcript data for their assessments.

No automated consequential decisions

Koji does not trigger, recommend, or automate any consequential action based on its outputs. The system provides information; the institution decides how to act on it. This separation is fundamental to the platform's design and is not configurable.

Bias and fairness

Bias in AI systems is a serious concern, particularly in education where outputs can affect how institutions perceive student experience across different cohorts. We approach this honestly: no AI system can claim zero bias. What matters is how bias risks are identified, mitigated, and monitored.

Known risks

• Quality scoring and language. Scoring may disadvantage non-native speakers or students with non-standard communication styles if rubrics over-weight linguistic clarity relative to substantive content.
• Voice processing and accents. Speech-to-text components may perform differently across accents, dialects, and speech patterns, potentially leading to lower transcript quality for some demographic groups.
• Cultural framing. Student feedback norms vary across cultures. Some students may express criticism indirectly or use politeness conventions that AI scoring could misinterpret as satisfaction.

Mitigation measures

• Multi-language support across 30+ languages, reducing the barrier for non-English-speaking students to provide feedback in their preferred language.
• Quality rubrics explicitly weighted toward content and substance rather than linguistic sophistication or grammatical precision.
• Regular bias audits that compare scoring distributions across demographic cohorts to detect systematic disparities.
• Ongoing monitoring for discriminatory outcomes in production data, with findings fed back into the risk management cycle.

Limitations

Bias testing is continuous and iterative. We do not claim to have eliminated bias from the system. New biases may emerge as the platform is deployed across different institutional contexts, student populations, and cultural settings. We are committed to detecting and addressing these as they arise, and we report findings transparently to deploying institutions.

No-training commitment

Koji does not use student data to train, fine-tune, or otherwise improve AI models. This commitment applies to:

• Conversation transcripts
• Voice recordings
• Quality scores
• Any derived or aggregated data

This commitment is not merely a policy statement. It is contractually documented in the Data Processing Agreement between Koji and the deploying institution.

University-controlled LLM infrastructure

Because each university connects Koji to their own enterprise LLM accounts, the LLM provider is not a Koji sub-processor. The institution's existing enterprise agreement with their AI provider governs data handling, retention, and model training restrictions. This architecture gives the university full visibility and control over the AI data processing chain. Where Koji provides an EU-native LLM alternative, the provider is added to the sub-processor register with the controller's prior consent.

AI literacy

Article 4 of the EU AI Act requires providers and deployers to ensure that staff and other persons dealing with AI systems have a sufficient level of AI literacy.

Staff training

Koji provides training materials for university staff operating the platform. These materials cover the purpose and function of each AI component, the meaning and limitations of quality scores and summaries, and how to interpret AI outputs in the context of institutional decision-making.

Student-facing information

Students participating in Koji interviews receive clear information explaining that they are speaking with an AI system, what the conversation will be used for, how their responses will be processed, and what rights they have regarding their data.

Documentation of capabilities and limitations

Platform documentation explicitly states what the AI can and cannot do. This includes known limitations such as the potential for hallucination in summaries, the possibility of scoring bias, and the fact that AI outputs should be treated as one input among many in institutional quality processes. We believe institutions make better decisions when they understand the boundaries of the tools they use.

Accuracy and robustness

Article 15 requires high-risk AI systems to achieve an appropriate level of accuracy, robustness, and cybersecurity. Koji implements the following measures.

Output accuracy

• Source tracing. All summarised claims in quality reports are linked to the specific transcript segments from which they were derived, enabling human verification.
• Human-coded benchmarks. Quality scoring is periodically validated against human-coded benchmarks to measure alignment between AI-generated scores and expert human judgement.
• Hallucination detection. Automated checks flag generated text that cannot be traced to source data, reducing the risk of fabricated claims appearing in quality reports.

Robustness against adversarial inputs

• Prompt injection protections. System prompts and input handling are designed to resist attempts by users to override AI behaviour through adversarial inputs.
• Input sanitisation. All user inputs are sanitised before processing to prevent injection attacks and ensure the integrity of AI outputs.

These measures reduce but do not eliminate the risk of inaccurate or manipulated outputs. Institutions should treat AI outputs as analytical aids and apply professional judgement when incorporating them into quality processes.

Questions about AI governance

For questions about our EU AI Act classification, risk management documentation, or any aspect of AI governance not covered on this page, contact us at compliance@koji.so or schedule a call with the team. We are happy to provide additional documentation, participate in institutional AI impact assessments, and support your internal governance review process.