AI Security and the Law: What the EU AI Act, NIST AI RMF, and ISO 42001 Actually Require of Builders

Published Jul 7, 2026 · 22 min read

For most of this blog’s life, AI security has been framed as a technical problem: attackers probe model boundaries, defenders patch them, researchers publish new attacks faster than defenses mature. That framing isn’t wrong — but it’s incomplete. In 2024–2026, AI security became a legal problem. Organizations that ship high-risk AI systems now face statutory obligations to adversarially test them, report security incidents within hours, and maintain documented risk management systems that would survive regulatory scrutiny.

This post maps three frameworks — the EU AI Act (Regulation (EU) 2024/1689), the NIST AI Risk Management Framework 1.0 (NIST.AI.100-1), and ISO/IEC 42001:2023 — to the attack classes covered in detail elsewhere on this blog. The goal is to answer a question practitioners increasingly face: “Our RAG system has a prompt injection vulnerability. Which regulation do we violate if we don’t fix it, and what exactly does fixing it require?”

The Compliance Imperative: Why Security Is Now a Legal Obligation

Three forces converged to make AI security a regulatory matter.

Incident severity crossed the materiality threshold. AI system compromises in healthcare diagnostics, financial decisioning, and critical infrastructure now cause harms that regulators treat as product liability events. The framing shifted from “software bug” to “defective product.”

Voluntary frameworks weren’t moving fast enough. The NIST AI RMF debuted in January 2023 as an explicitly voluntary framework. Within months, US federal procurement policy began treating it as a de facto requirement for AI acquisition. The gap between “voluntary” and “mandatory” narrowed to policy language.

The EU moved first, forcing global compliance. Regulation (EU) 2024/1689 — the EU AI Act — was published in the Official Journal of the European Union on July 12, 2024 (OJ L 2024/1689). It applies directly in all EU member states without national implementation legislation. Because it applies to any AI system deployed in the EU regardless of where the developer is located, it effectively became a global standard for any organization with EU market exposure.

The EU AI Act: Risk Architecture and the Security Articles That Matter

The EU AI Act (Regulation (EU) 2024/1689) establishes a risk-tiered regulatory structure. Understanding which tier your system falls into determines which security obligations apply.

Risk Classification

Unacceptable risk (prohibited): Systems are banned outright — AI-enabled biometric categorization based on sensitive attributes, social scoring by public authorities, real-time remote biometric identification in public spaces with limited exceptions (Article 5). No security obligation applies because the deployment itself is prohibited.

High-risk AI systems (Annex III): Systems in eight domains including biometric identification, critical infrastructure management, education, employment, essential private and public services, law enforcement, migration, and administration of justice (Article 6 and Annex III). These systems bear the full weight of the Act’s security requirements.

General-purpose AI models (GPAI, Articles 51–56 / Title VIII): AI models trained on large amounts of data using self-supervision at scale, capable of serving a wide range of downstream tasks. “Systemic-risk” GPAI models — those where Article 51(1) applies, with training compute exceeding 10^25 FLOPs as a presumption (rebuttable) of systemic risk — face heightened obligations including mandatory red-team evaluations.

Limited and minimal risk: Chatbots and image generators face transparency obligations (Article 50) but not the security mandates discussed here.

Article 9: The Mandatory Risk Management System

Article 9 requires providers of high-risk AI systems to establish, implement, document, and maintain a risk management system throughout the system lifecycle. This is the architectural security requirement.

Specifically, Article 9(2) requires the risk management system to:

Identify and analyze known and reasonably foreseeable risks associated with the AI system
Estimate and evaluate the risks that may emerge when the system is used as intended and under conditions of reasonably foreseeable misuse
Evaluate risks in light of data from post-market monitoring

Article 9(4) requires that “appropriate and targeted risk management measures” be adopted for residual risks. Article 9(5) requires that high-risk AI systems be tested to identify appropriate risk management measures and to verify compliance — and explicitly states that testing shall be performed against preliminarily defined metrics and probabilistic thresholds.

For security purposes, this means adversarial testing — including tests for prompt injection, data poisoning susceptibility, model extraction resistance, and robustness to distribution shift — is not optional for high-risk AI systems. It is a statutory requirement.

Article 15: Accuracy, Robustness, and Cybersecurity

Article 15 is the most technically specific security provision in the Act. It requires that high-risk AI systems be designed and developed to achieve, throughout their lifecycle, an “appropriate level of accuracy, robustness and cybersecurity, and to perform consistently in those respects.”

Article 15(3) addresses technical robustness: high-risk AI systems must be designed to be resilient against unauthorized third-party attempts to alter their use, outputs, or performance by exploiting system vulnerabilities — encompassing adversarial attacks, data poisoning attacks, and model poisoning attacks as described in the recitals.

Article 15(4) addresses resilience against errors, faults, and inconsistencies that may occur in high-risk AI systems, ensuring that such systems can adequately cope with these and continue operating at an appropriate level of performance.

The practical interpretation: a high-risk AI system that is demonstrably vulnerable to documented attack classes — and that has not been tested for and mitigated against those vulnerabilities — will struggle to demonstrate the “appropriate level” of robustness that Article 15 requires. The Act uses a risk-based standard rather than an absolute one, so the adequacy of security measures is assessed in proportion to the risk; but the burden falls on providers to demonstrate that level has been met.

Articles 51–56: GPAI and Systemic-Risk Models

General-purpose AI models with systemic risk — where Article 51(1)‘s 10^25 FLOPs training compute threshold applies as a presumption — face obligations that go beyond the high-risk framework. Article 55 requires providers of systemic-risk GPAI models to:

Perform model evaluations including adversarial testing (Article 55(1)(a)) — described in the Act as testing “to identify and mitigate systemic risks”
Track, document, and report serious incidents to the AI Office without undue delay
Apply cybersecurity protection adequate to the level of risk

The AI Office’s Code of Practice for GPAI Models is the implementing guidance for these provisions. The Code of Practice was finalized by the European AI Office in July 2025 and provides specific operational requirements for providers of GPAI models, including red-teaming methodologies and incident reporting protocols. Organizations should consult the finalized text at ai-office.ec.europa.eu for the current authoritative version.

Incident Reporting

Article 73 establishes incident reporting obligations for providers of high-risk AI systems. Serious incidents must be reported to national market surveillance authorities without undue delay, and in any case within the following tiered deadlines from first becoming aware:

Two days (calendar): incidents involving widespread infringement (cross-border fundamental-rights breaches, as referenced in Article 73) or serious and irreversible disruption of critical infrastructure management or operation
Ten days: incidents resulting in death
Fifteen days: other serious incidents (the default ceiling)

For systemic-risk GPAI models, Article 55(1)(c) requires reporting of serious incidents to the AI Office without undue delay.

Important: Always verify Article 73’s specific deadlines against the final adopted text of Regulation (EU) 2024/1689 published in the Official Journal, as implementing guidance from national authorities may add precision. The deadlines above are drawn from the Act’s tiered structure; the GPAI-specific obligation in Article 55 has its own formulation.

Enforcement and Penalties

Infringements of specific provisions carry different penalty ceilings:

Violations of the prohibited practices in Article 5: up to €35,000,000 or, for undertakings, up to 7% of total worldwide annual turnover, whichever is higher (Article 99(3))
Violations of other obligations applicable to providers (including the Article 9 and 15 security requirements): up to €15,000,000 or 3% of total worldwide annual turnover, whichever is higher (Article 99(4))
Incorrect, incomplete, or misleading information to authorities: up to €7,500,000 or 1% of total worldwide annual turnover, whichever is higher (Article 99(5))

These figures come directly from Regulation (EU) 2024/1689 Article 99 as published. The penalties are ceilings — national enforcement authorities exercise discretion — but they signal regulatory seriousness.

Application timeline: The Act entered into force on August 1, 2024. Provisions on prohibited AI practices (Article 5) applied from February 2, 2025. Obligations for GPAI models applied from August 2, 2025. The high-risk AI system requirements for systems listed in Annex III apply from August 2, 2026. High-risk AI systems covered via the Annex I product-safety route (systems embedded in regulated products such as medical devices and machinery) follow a longer transition timeline. This means the full Article 9 and Article 15 obligations come into force for most Annex III high-risk systems in August 2026 — imminently at time of writing.

NIST AI RMF: From Voluntary Framework to Procurement Requirement

The NIST AI Risk Management Framework 1.0 (NIST.AI.100-1) was published by the National Institute of Standards and Technology in January 2023. It is organized around four core functions.

The Four Core Functions

GOVERN: Establishing organizational policies, processes, and accountability structures for AI risk. Govern is the enabling function — an organization with no AI governance structure cannot meaningfully execute the other three. For security, Govern includes designating responsibility for adversarial testing, defining the risk appetite for AI system deployment, and establishing the policies that determine when a discovered vulnerability triggers incident response.

MAP: Identifying and categorizing AI risks in context. Map involves understanding what the AI system does, who uses it, what could go wrong, and how that harm would manifest. For security practitioners, Map is the threat modeling phase: which attack classes are plausible given this deployment context?

MEASURE: Analyzing and assessing identified risks using appropriate metrics and tools. Measure is where security testing lives. The AI RMF Playbook — the companion document to NIST.AI.100-1 — specifies practices at the sub-function level.

MANAGE: Prioritizing and addressing risks identified in the Measure phase. Manage includes both mitigation implementation and the ongoing monitoring required to detect new risks as the system and its environment evolve.

The Playbook’s Security-Specific Practices

The AI RMF Playbook identifies specific suggested practices for each sub-function. The most security-relevant:

MEASURE 2.5 — Addresses AI system validity and reliability in deployment context — whether the system performs as expected and continues to perform consistently over time. For security, MEASURE 2.5 maps to monitoring for behavioral drift that could indicate model tampering or environmental shifts that degrade robustness controls.

MEASURE 2.6 — Addresses AI system safety risks, including safe-to-fail design and safe operation under anomalous conditions. This practice covers the question of what happens when an AI system encounters conditions outside its designed operating envelope — relevant to adversarial inputs that push the system into unexpected behavior states.

MEASURE 2.7 — Specifically addresses AI risk and impact assessment methods for adversarial attacks. The Playbook suggests: conduct adversarial testing including red-teaming to identify AI system vulnerabilities, attack surfaces, and exposures, and document results. This practice is the NIST equivalent of the EU AI Act’s Article 9(5) testing mandate.

MANAGE 4.1 / 4.2 — Residual risk treatment and incident response. The AI RMF’s relationship to existing NIST cybersecurity guidance is explicit: NIST SP 800-61r3 (Incident Response) remains the baseline; MANAGE 4.x extends the lifecycle to AI-specific incidents including model compromise, behavioral drift, and adversarial attack discovery post-deployment.

Relationship to NIST CSF 2.0

NIST Cybersecurity Framework 2.0 (published February 2024) added a sixth function — Govern — mirroring the AI RMF’s structure. The intent is explicit alignment: organizations using CSF 2.0 for their overall security program should integrate AI RMF practices into the same governance structure rather than maintaining separate frameworks. The AI RMF’s MAP function maps roughly to CSF’s Identify; MEASURE to Identify and Detect; MANAGE to Protect, Respond, and Recover.

Federal Adoption and Procurement

Executive Order 14110 (October 2023) directed federal agencies to develop standards and guidance for AI safety and security based on NIST’s work. The Office of Management and Budget Memorandum M-24-10 (March 28, 2024) established AI governance requirements for federal agencies using AI. M-24-10 explicitly references the NIST AI RMF and directs agencies to use it for risk management of covered AI — making the framework functionally expected within federal agency operations. For vendors selling AI systems to the US federal government, alignment with the AI RMF has become an increasingly common contractual expectation, even where the framework itself remains technically voluntary.

ISO/IEC 42001:2023: What an AI Management System Must Document on Security

ISO/IEC 42001 — “Artificial intelligence — Management system” — was published in December 2023. It follows the Annex SL high-level structure familiar from ISO 27001 and ISO 9001, making it integrable with existing management system certifications.

What 42001 Requires

ISO/IEC 42001 requires organizations to establish, implement, maintain, and continually improve an AI management system (AIMS). The core security-relevant requirements:

Clause 6 (Planning): The organization must determine AI-related risks and opportunities. For security, this means identifying adversarial threats as part of the risk assessment process — not just operational risks like accuracy degradation, but deliberate attacks.

Clause 8 (Operation): The AIMS must include operational controls for AI system development and deployment. Annex A (controls) and Annex B (implementation guidance) specify that organizations should consider adversarial robustness as part of the AI system design review.

Clause 9 (Performance evaluation): The organization must monitor and measure AI system performance, including security properties. Where risks have been identified, controls must be verified as effective.

Clause 10 (Improvement): Nonconformities — including discovered security vulnerabilities — must trigger corrective action processes.

The standard does not specify which adversarial tests must be performed; it requires that the organization’s risk assessment process identify applicable threats and that appropriate controls be implemented and verified. For an organization that has identified prompt injection as a relevant risk for their GPAI deployment, the AIMS must document the control (prompt filtering, system prompt isolation, output monitoring) and verify its effectiveness.

Relationship to ISO 27001

ISO/IEC 27001 (information security) and ISO/IEC 42001 (AI management system) are designed to coexist. An organization with ISO 27001 certification extends its Information Security Management System to include AI-specific controls via a combined ISMS+AIMS. ISO 42001 is referenced in various regulatory and industry contexts as a relevant implementation framework for AI governance, though its specific relationship to EU AI Act compliance obligations should be verified against current AI Office guidance and any harmonized standards adopted under the Act.

Cross-Reference Map: Attack Class → Regulation → Blog Post

The following table maps attack classes to their regulatory anchors and the technical detail available elsewhere on this blog.

Attack class	EU AI Act article	NIST AI RMF practice	ISO 42001 clause	Blog post
Prompt injection	Art. 15 (robustness against adversarial manipulation)	MEASURE 2.7	Clause 8 (operational controls)	Prompt Injection via Role Confusion, Indirect Prompt Injection Incidents Survey
Indirect prompt injection	Art. 15 (adversarial manipulation); Art. 9 (risk mgmt)	MEASURE 2.7, MEASURE 2.6	Clause 8	Indirect Prompt Injection Incidents Survey, Copilot File Exfiltration via Prompt Injection
Training data poisoning	Art. 9 (risk mgmt system); Art. 15 (robustness against data poisoning)	MEASURE 2.7, MAP 2.3	Clause 6 (risk planning)	Pretraining Corpus Poisoning, RAG Memory Poisoning
Backdoor / trojan attacks	Art. 9 (risk mgmt); Art. 15 (model poisoning resilience)	MEASURE 2.7	Clause 8	Backdoor Attacks in Foundation Models, Sleeper Agents
Model extraction	Art. 15 (cybersecurity); Art. 9 (risk mgmt)	MEASURE 2.7, MEASURE 2.6	Clause 8	Model Extraction via API Queries
Membership inference	Art. 10 (data governance); GDPR intersection	MEASURE 2.10	Clause 6	Membership Inference Attacks
Supply chain attacks	Art. 9 (risk mgmt); Art. 25 (obligations of importers/distributors)	MANAGE 4.1, MEASURE 2.7	Clause 8	AI Agent Supply Chain Attacks, LLM Router Supply Chain Attack
Adversarial examples	Art. 15 (robustness against adversarial input)	MEASURE 2.7, MEASURE 2.6	Clause 8	Adversarial Attacks on Vision-Language Models, Adversarial Examples (Foundational)
Privacy / data exfiltration	Art. 10 (data governance); Art. 15; GDPR intersection	MEASURE 2.10	Clause 6, Clause 8	RAG Privacy Attacks, Gradient Inversion Attacks

The EU AI Act explicitly preserves the application of the GDPR (Regulation 2016/679) and does not replace it. For AI systems that process personal data — which includes most deployed LLMs in commercial contexts — both the AI Act’s security requirements and GDPR’s data protection obligations apply concurrently. Privacy attacks (membership inference, training data reconstruction, gradient inversion) implicate both frameworks: Article 15 of the AI Act (robustness) and Article 32 of the GDPR (security of processing).

Red-Teaming as a Regulatory Requirement

The most significant doctrinal shift in the 2024–2026 regulatory wave is the treatment of red-teaming. Prior to 2024, red-teaming for AI systems was understood as a security best practice — valuable, recommended, but not legally required. The EU AI Act changed this.

What the EU AI Act Requires

Article 9(5) requires that high-risk AI systems be tested to identify appropriate risk management measures and to verify compliance. Article 55(1)(a) requires that providers of systemic-risk GPAI models perform model evaluations including adversarial testing to identify and mitigate systemic risks.

The Act does not define “adversarial testing” with the specificity of a technical standard. The AI Office’s Code of Practice for GPAI (finalized July 2025) provides implementing guidance for GPAI model providers, including red-teaming requirements. For high-risk AI systems in Annex III categories, standards bodies (including ENISA and CEN-CENELEC) are developing harmonized standards that will provide a presumption of conformity. Organizations that comply with a harmonized standard will be presumed to satisfy the corresponding legal requirement.

Until those standards are finalized, the practical obligation is: document your adversarial testing methodology, conduct testing proportionate to the risk, and maintain records that demonstrate the testing occurred and that residual risks were evaluated.

NIST’s Framing

NIST AI RMF Playbook practice MEASURE 2.7 describes red-teaming in operational terms: conduct adversarial testing including red-teaming to identify AI system vulnerabilities, attack surfaces, and exposures, and document results. The NIST companion document Adversarial Machine Learning: A Taxonomy and Terminology (NIST.AI.100-2e2025) provides technical vocabulary for these evaluations. This document was finalized in 2025; earlier iterations were published as drafts for public comment.

For organizations aligning with the AI RMF, MEASURE 2.7 is the practice that requires red-teaming. The word “suggested” in the Playbook’s practice descriptions reflects the framework’s voluntary character at the federal level — but as noted above, voluntary at the statutory level does not mean optional in procurement contexts.

Operationalizing the Requirement

A red-teaming program that satisfies regulatory scrutiny needs to answer three questions the AI Act’s Article 9 implicitly requires:

What are the plausible attack scenarios for this system in its deployment context? (The MAP function in NIST terms; risk identification in Article 9(2)(a) terms.)
Were those scenarios tested, by whom, with what methodology, and what were the results? (Article 9(5) testing documentation; MEASURE 2.7 documentation.)
What residual risks remain after controls are applied, and why were they accepted? (Article 9(2)(c) risk evaluation; MANAGE function in NIST terms.)

The MITRE ATLAS framework provides the most systematic taxonomy for answering the first question — mapping attacker tactics and techniques to system components. The AI Incident Response Playbook addresses what happens when pre-deployment red-teaming fails to catch everything.

Incident Reporting Obligations

EU AI Act Reporting

Article 73 establishes the incident reporting timeline for providers of high-risk AI systems. Serious incidents must be reported to national market surveillance authorities without undue delay, within the following tiered deadlines from first becoming aware:

Two calendar days: Incidents involving widespread infringement (cross-border fundamental-rights breaches, as referenced in Article 73) or serious and irreversible disruption of the management or operation of critical infrastructure
Ten calendar days: Incidents resulting in death
Fifteen calendar days: Other serious incidents (the default)

For systemic-risk GPAI models, Article 55(1)(c) requires reporting of serious incidents to the AI Office without undue delay.

“Serious incident” is defined in Article 3(49) as an incident — or near-miss — that has led, or may lead, to: death or serious harm to health; disruption of the management or operation of critical infrastructure; breach of obligations protecting fundamental rights; serious harm to property or the environment; or, for GPAI models, other significant impacts as relevant. In security terms, this captures: a model compromise that enables extraction of health records (GDPR + AI Act), an adversarial attack against a high-risk AI used in loan decisioning that produces discriminatory outputs at scale, or a backdoor activation in a system used for critical infrastructure management.

Always verify these deadlines against the final adopted text of Regulation (EU) 2024/1689 and national market surveillance authority guidance — implementing acts may add precision, and the tiering is complex.

Interaction with Existing IR Programs

The EU AI Act’s reporting obligations stack with:

GDPR Article 33: 72-hour notification to supervisory authority for personal data breaches
NIS2 Directive: 24-hour early warning, 72-hour incident notification, and 1-month final report for significant cybersecurity incidents affecting essential services
DORA (Digital Operational Resilience Act): Incident reporting for financial sector entities

For organizations in regulated industries deploying high-risk AI, a single incident may trigger concurrent obligations under multiple regimes. The AI Act reporting goes to national market surveillance authorities (the bodies designated by each member state under Article 70); GDPR reporting goes to data protection authorities (DPAs); NIS2 reporting goes to national cybersecurity authorities (e.g., BSI in Germany, ANSSI in France).

Practical implication: incident response playbooks for EU-deployed high-risk AI systems need a regulatory notification workflow that is distinct from the technical response workflow, that maps each incident type to its applicable reporting requirements, and that has pre-approved notification templates to meet the tiered Article 73 deadlines (2 / 10 / 15 calendar days by incident severity, plus GDPR’s 72h for data breaches and NIS2’s 24h early warning) without losing time to legal review during the crisis. The AI Incident Response Playbook covers the technical response phases; the regulatory notification layer is an organizational process that sits alongside it.

What to Do Now: A Compliance Readiness Checklist

Not every organization needs to comply with every requirement above. The obligations are conditional on system type, deployment context, and geography. This checklist helps prioritize:

1. Classify your AI systems under the EU AI Act risk tiers, and determine your role (provider vs. deployer). If you have EU market exposure and deploy AI systems, determine whether any fall into Annex III categories. Critically, note that Articles 9 and 15 impose their most demanding obligations on providers (those who develop AI systems or place them on the market), not on deployers who merely use third-party systems. Deployers have their own obligations (Article 26) but are not responsible for the provider-level technical compliance. Understanding your role in the AI supply chain is the prerequisite for determining which obligations bind you.

2. Assess whether any models qualify as GPAI or systemic-risk GPAI. If you are a foundation model provider (not a deployer of third-party models), determine whether any model meets the systemic-risk threshold under Article 51(1). Exceeding 10^25 FLOPs of training compute creates a rebuttable presumption of systemic risk; the AI Office may also designate other models. If you are deploying third-party models, understand what the model provider’s obligations are and how those flow through supply-chain contracts.

3. Conduct and document adversarial testing for each high-risk system. At minimum, test against the attack classes in the cross-reference table above that are relevant to your deployment context. Document the methodology, who performed the testing, the date range, the findings, and the residual risks. This documentation is what an Article 9(5) compliance audit will request.

4. Establish incident response with regulatory notification workflows. Map your AI incident types to their reporting obligations. Identify which authority receives which notification, pre-draft notification templates, and ensure the tiered Article 73 deadlines (2 / 10 / 15 calendar days by incident severity) are built into your incident response timeline. If you are also subject to GDPR (72h for data breaches) and/or NIS2 (24h early warning, 72h notification), consolidate the notification map — a single AI security incident may trigger multiple concurrent reporting obligations to different authorities.

5. Align with NIST AI RMF to satisfy US federal procurement and build the governance layer. Whether or not you have direct federal contracts, AI RMF alignment produces the documentation artifacts (risk assessments, testing records, governance policies) that also support EU AI Act compliance. The four functions — Govern, Map, Measure, Manage — are a useful organizing structure for building an AI security program that scales across regulatory jurisdictions.

Primary Sources and Further Reading

All regulatory claims in this post are drawn from primary source text. Readers implementing compliance programs should verify against the authoritative texts:

NIST.AI.100-2e2025 (Adversarial Machine Learning: Taxonomy and Terminology of Attacks and Mitigations): Finalized in 2025. Available at nist.gov. Earlier pre-decisional draft iterations were published in 2023 and 2024; use the finalized version for compliance citations.
EU AI Act (Regulation (EU) 2024/1689): Published in the Official Journal of the European Union, L series, 2024. The text is available via EUR-Lex. Article numbers cited here refer to the final adopted text.
NIST AI RMF 1.0 (NIST.AI.100-1): Published January 2023. Available at csrc.nist.gov/pubs/ai/100/1/final.
NIST AI RMF Playbook: Companion document to NIST.AI.100-1, providing suggested practices for each core function and sub-function. Available at airc.nist.gov.
ISO/IEC 42001:2023: Available through national standards bodies (ANSI in the US, BSI in the UK, DIN in Germany) and directly from ISO. The standard is not freely available; it is a purchased publication.
European AI Office: Official source for Code of Practice for GPAI models (finalized July 2025) and GPAI model guidance — ai-office.ec.europa.eu.

Regulatory text evolves. If you are reading this post more than six months after its publication date (July 2026), verify that the cited article numbers, deadlines, and penalty structures remain current against the authoritative source.