Serious Incident Reporting and Post-Market Monitoring

Pre-deployment evaluation tests a system against the situations somebody thought of. Deployment tests it against the situations nobody thought of, at a volume no eval suite can match, and the only way that information gets back into the assurance argument is if someone is obliged to collect and report it. This is the half of assurance that happens after the launch blog post.

The two obligations, and the clocks

European law splits the duty in two. Under Article 72, a provider of a high-risk system must establish and document a post-market monitoring system proportionate to the risks, running on a written plan that forms part of the technical documentation, and must actively collect and analyse performance data over the system's whole life, including data from deployers (EU AI Act, Article 72). Monitoring is a standing capability, not a reaction.

Article 73 handles the reactive part. A serious incident must be reported to the market surveillance authority immediately once a causal link to the AI system is established or reasonably likely, and in any event within deadlines that step down with severity: fifteen days in the general case, ten days where a person has died, and two days where the incident involves a widespread infringement or a serious and irreversible disruption to critical infrastructure (EU AI Act, Article 73). An incomplete initial report followed by a complete one is explicitly permitted, which tells you the drafters preferred early notice to tidy notice.

Two details drive the engineering. First, the clock starts on awareness, not on resolution, so the detection pipeline is on the critical path of compliance and a monitoring system that surfaces an anomaly a month late has already blown the deadline. Second, providers of general-purpose models with systemic risk carry a parallel duty under Article 55(1)© to keep track of, document and report serious incidents to the AI Office without undue delay (EU AI Act, Article 55), which is a different recipient and a different trigger from the Article 73 route.

Incident, hazard, near miss

The OECD's definitional work separates an AI incident, where harm actually occurred, from an AI hazard, where an event was potentially harmful but harm did not materialise (OECD, AI Incidents and Hazards Monitor methodology). The distinction matters because hazards are far more numerous, far cheaper to learn from, and the category most reporting regimes fail to capture.

Aviation solved this long before software. The NASA Aviation Safety Reporting System takes voluntary reports from pilots and controllers, strips identifying detail before the report enters the database, and is paired with an FAA policy under which a timely report about an inadvertent violation is treated as evidence of a constructive attitude rather than as a confession; 14 CFR 91.25 bars the use of ASRS reports in enforcement, excepting accidents and criminal offences. The result is volume: the programme's own briefing puts 2023 intake at 106,087 reports against a cumulative total of over two million (NASA, ASRS Program Briefing).

The lesson is structural. You get near-miss data when reporting is confidential and non-punitive, and you do not get it when a report is an admission of liability. An internal AI incident process that routes to legal before it routes to engineering will be told about outages and never about the time an agent nearly wired the money.

When it breaks

Every incident corpus undercounts, and the direction of the bias is known. The OECD monitor builds from international media reporting, and its own methodology notes that this is a starting point because many incidents are never publicly disclosed. Counting press coverage measures press coverage. Published trend lines, including the widely quoted rise in monthly reported incidents between 2022 and 2025, are consistent with more AI, more harm, more journalists on the beat, or all three, and the data cannot separate them.

Attribution is the hard part of the definition. "Serious incident" presumes you can tie a harm to the system. For a model that contributed one step of advice inside a human decision, the causal link is a judgement call made under a fifteen-day clock, and the incentive on the person making it is obvious.

There is no AI equivalent of a CVE identifier. Security has a shared namespace, a severity scale and a disclosure norm; AI has several incompatible databases, no stable identifier, and no agreed severity taxonomy, so two organisations reporting the same failure produce records that cannot be joined.

Monitoring without a feedback edge into the safety case is theatre. The point of post-market data is to invalidate claims the pre-deployment case made. If the incident process terminates in a ticket queue rather than in a defeater checked against the argument, the assurance document stays frozen at launch while the evidence moves underneath it.