How the EU's Code of Practice Advances AI Safety
The Code provides a powerful incentive to push frontier developers toward measurably safer practices.
.jpg)
The European Union just published the finalized Code of Practice for general-purpose AI models, transforming the AI Act's high-level requirements into concrete standards that will likely shift frontier AI companies' practices toward safer ones.
Among the Code’s three chapters (copyright, transparency, and safety & security), requirements outlined in the Safety and Security section mark particular advances in frontier AI safety. The chapter — drafted by chairs Yoshua Bengio, Marietje Schaake, and Matthias Samwald, along with six vice chairs — targets general-purpose models deemed to pose systemic risks, currently defined as those trained with more than 10^25 floating-point operations (FLOPs).
The 10^25 threshold may evolve. The 10^25 threshold captures all of today's frontier models. This threshold can be adapted as the technology evolves. The Code emerged from an extensive consultation process with over a thousand stakeholders (from industry, academia, and civil society) providing feedback across multiple rounds.
Companies have a strong incentive to adopt the Code. Companies who follow the Code gain a “presumption of conformity” with Articles 53 and 55 of the AI Act, which comes into force on August 2, 2025. Presumption of conformity means that regulators will assume a company is compliant with the law if it follows the Code's standards. This incentive overwhelmingly compelled firms to adopt previous EU Codes, such as the Code of Practice on Disinformation. Indeed, some companies like OpenAI have already stated that they will comply with the Code, recognizing its value for demonstrating regulatory compliance.
The Code requires a comprehensive risk management process. Companies must predetermine what levels of risk are acceptable and maintain risks below these thresholds through risk assessment and mitigation. They must document their compliance in two key documents: a Framework (similar to existing Frontier AI Safety Policies) and a Model Report, which shows application of the Framework for each model (similar to model cards).
The process follows a clear cycle: identify risks, analyze the current risk levels, determine if these are acceptable, and if not, implement mitigations and repeat until risks reach acceptable levels.
CBRN, Cyber, Loss of Control, and Manipulation. The risk identification process must account for multiple sources of risk, including the model’s capabilities, its propensities (e.g., its tendency to hallucinate), and its affordances (e.g., its access to tools). This expands the scope of current identification practices by requiring companies to consider specific types of risks.
The Code identifies four systemic risks by default:
This marks a significant improvement, as currently, no company frameworks comprehensively address all these risk categories. For instance, Anthropic's Responsible Scaling Policy lacks comprehensive risk management for cyber offense risks and doesn't address manipulation risks at all, while Meta’s Framework omits loss-of-control risks entirely.
Risks from “Risk Identification” must then be analyzed. Risk analysis must include:
Want to contribute to the conversation? Pitch your piece.
Risk modeling and estimation represent a major advancement over current practices.
Risk modeling, defined in the Code as “a structured process aimed at specifying pathways through which a systemic risk stemming from a model might materialize,” is a foundational part of the risk assessment process. Risk modeling, often overlooked in current company practices, informs the model evaluations and mitigations required by making explicit how risks may arise from a model.
Risk estimation requires companies to explicitly specify “the probability and severity of harm for the systemic risk” using quantitative, semi-quantitative, or qualitative formats. Current practices typically rely solely on comparing model evaluation results against capability thresholds and focus only on estimating model capabilities rather than estimating actual risks.
Companies must also preemptively define risk tiers that are both measurable and defined in terms of model capabilities. This means organizations will have to decide in advance which levels of capabilities would be unacceptable, rather than use their discretion around the time of model deployment.
Public transparency requirements. Companies must publish summarized versions of their Model Reports and Frameworks, including “high-level descriptions of the systemic risk assessment results and the safety and security mitigations implemented.” This enables public scrutiny and peer review of safety practices. While full transparency (with dangerous information redacted) would be ideal, these summarized reports are a significant improvement over current practices. Several companies have yet to release any safety frameworks, model cards are inconsistently provided, and many omit safety evaluations entirely.
External evaluation requirements. “In addition to internal model evaluations, Signatories will ensure that adequately qualified independent external evaluators conduct model evaluations.” This requirement addresses longstanding concerns about companies grading their own homework on safety.
Security plan. Companies must outline a “Security Goal” to help assess whether security measures are adequate. This goal must specify which threat actors their security mitigations are intended to protect against, including non-state external threats, insider threats, and other expected threat actors.
Incident reporting system. Companies must track and communicate serious incidents to the European Commission and national authorities, providing regulators with real-time visibility into the evolving risk landscape. “Serious incidents” include deaths, severe harm to physical or mental health, irreversible disruption of critical infrastructure, and major cybersecurity breaches such as model weight exfiltration or cyberattacks. Initial reports must be filed within two to 15 days, depending on severity, with follow-ups required until the incidents are resolved.
The Code does not include internally deployed models. The AI Act, for which the Code provides compliance guidance, only regulates models placed on the EU market. Models trained and used exclusively for internal AI research and development remain outside the Code's scope. This is a crucial gap for loss-of-control scenarios, where internally deployed models are a significant source of risk.
/odw-inline-subscribe-cta
Companies retain discretion. Companies can define key elements of their own risk management approach, such as how they categorize risk levels or decide what counts as acceptable risks. While standardized thresholds set by regulators would provide more consistency, such prescriptive approaches remain challenging to define for all risks, given the nascent state of AI risk management practices.
However, future regulations should work toward establishing explicit acceptable risk thresholds, following the model of other high-risk industries. For instance, the Federal Aviation Administration sets the acceptable frequency of catastrophic accidents at less than one per billion flight hours, equivalent to one catastrophic event every 114,155 plane years (where catastrophic accidents are defined as “failure conditions which would prevent continued safe flight and landing”).
Enforcement limitations. The US government has already pushed back against the Code, while companies are also increasing the pressure. A recent example is a letter from 46 tech CEOs asking for a two-year pause on obligations for general-purpose AI models. (Some companies, however, have stated that they will comply with the Code, such as OpenAI.) Therefore, the success of the Code will largely hinge on how much political will and resources the European Commission dedicates to enforcement.
Financial penalties limitations. Finally, it is not clear if the financial penalties of “up to 3% of global annual turnover or €15 million, whichever is higher,” for violations of the Code will be enough to deter non-compliance. Many frontier AI companies currently have expenses far exceeding their revenues, potentially making these fines a manageable cost of doing business.
Overall, the Code of Practice represents a massive step in the right direction to make AI development safer. From a risk management standpoint, it meaningfully improves upon current industry practices. By requiring explicit risk modeling, operationalizing external evaluations, and mandating public transparency, the Code addresses many of the weaknesses of existing voluntary frameworks.
Its influence will likely extend beyond Europe. Companies operating globally may find it easier to implement these standards universally rather than maintaining separate processes for different markets. Moreover, new regulations often draw inspiration from existing frameworks. While the Code is not perfect, it provides a blueprint for AI safety regulations worldwide and establishes a regulatory floor for future policies to build upon. Both European and global regulations should expand on this foundation to address the limitations outlined above. Additionally, while formal regulation is important, voluntary commitments from companies have historically played a valuable role in advancing safety practices across the field and should continue alongside these regulatory efforts.
Acknowledgment
The author would like to thank Su Cizem, Alejandro Tlaie Boria, Siméon Campos, and Lily Stelling for their valuable feedback on earlier drafts of this article. Any views and errors are the author's own.
The author, Henry Papadatos, contributed feedback to the Code of Practice as part of the public process that gathered input from over 1,400 stakeholders.
See things differently? AI Frontiers welcomes expert insights, thoughtful critiques, and fresh perspectives. Send us your pitch.
Abandoning liability mechanisms risks creating a dangerous regulatory vacuum.
Placing AI in a nuclear framework inflates expectations and distracts from practical, sector-specific governance.
