AI incident reporting is the systematic recording, sharing, and investigation of cases where AI systems cause harm or come close to it. The model is aviation. When an aircraft crashes, or nearly does, it triggers a rigorous investigation whose findings are published and fed back into design, training, and regulation across the whole industry. That discipline is a large part of why flying is as safe as it is. The field learns from every failure, once, on everyone's behalf.
AI has almost nothing like it. When a model causes harm, whether a discrete failure, a misuse that slips its guardrails, or a dangerous behaviour caught in testing, there is no consistent requirement to record it, no shared place to report it, and no standard process to investigate it. Lessons that could protect everyone stay locked inside one company, or vanish. Incident reporting proposes to fix that: a duty to report defined categories of AI harm and near-miss, into a shared system, so the field can learn.
What a good system would capture
The useful version is broader than obvious disasters.
- Deployed harms, where a system in the real world caused damage or serious malfunction.
- Near-misses, where something went wrong but harm was narrowly avoided, which aviation treats as just as informative as crashes.
- Dangerous behaviours found in testing, including the kinds of failures surfaced by red teaming and evaluations, so that a hazard found by one lab informs the rest.
- Security events, such as attempts to steal model weights or circumvent safeguards.
Reporting also has to be structured and, where appropriate, protected, so that organisations disclose honestly rather than hiding failures to avoid embarrassment or liability. Aviation learned to separate safety investigation from blame for exactly this reason.
Why it is worth doing
The benefits are real and, unusually for AI governance, uncontroversial. Shared incident data lets the field spot patterns no single organisation would see, gives regulators an evidence base grounded in what is actually going wrong rather than speculation, warns everyone about a failure mode as soon as one party encounters it, and builds, over time, an institutional memory the field currently lacks. It is one of the lower-cost, higher-consensus measures available, and the Foundation supports making it mandatory and well-designed.
The limit built into the idea
But notice the tense. Incident reporting is fundamentally retrospective. It works by learning from harms that have already happened, which is exactly why it made aviation safe: aircraft failures, however tragic, are survivable at the level of the industry, so each one can teach the fleet. The system improves crash by crash.
That logic breaks for the risks the Foundation is most concerned with. A catastrophic failure of a superintelligent system is not an incident you convene a board to investigate afterward. The whole argument for existential risk is that the most serious failure may be the one you do not get to learn from, because there is no after. Incident reporting handles the accumulating, survivable harms of AI extremely well and, by its nature, cannot address the unrecoverable one.
Learning from failures assumes you survive them. For the failures that matter most, that assumption is the whole problem.
So incident reporting belongs in the toolkit, and it cannot be the top of it. It should be built, mandated, and designed to protect honest disclosure, and it should sit alongside the anticipatory measures, the limits and verification set out in our plan, that exist precisely because some failures cannot be handled after the fact.
Common questions.
AI incident reporting is the systematic recording, sharing, and investigation of cases where AI systems cause harm or nearly do. It is modelled on aviation, where every crash and near-miss triggers an investigation whose findings are published and fed back into the whole industry. The aim is a duty to report defined categories of AI harm and near-miss into a shared system, so the field can learn from failures once, collectively, rather than having each lesson stay locked inside a single company or disappear.
More than obvious disasters. A good system records deployed harms where a real-world system caused damage, near-misses where harm was narrowly avoided, dangerous behaviours found in testing such as those surfaced by red teaming and evaluations, and security events like attempts to steal model weights or bypass safeguards. Reporting should be structured and, where appropriate, protected from blame, so organisations disclose honestly rather than hiding failures to avoid embarrassment or liability.
Because shared incident data lets the field detect patterns no single organisation would see, gives regulators an evidence base grounded in what is actually going wrong rather than speculation, warns everyone about a failure mode as soon as one party hits it, and builds an institutional memory the field currently lacks. It is one of the lower-cost, higher-consensus measures in AI governance, which is why making it mandatory and well-designed attracts broad agreement.
It is inherently retrospective: it works by learning from harms that have already happened. That is what made it so effective in aviation, where failures are survivable at the industry level so each one can teach the fleet. But that logic breaks for catastrophic risks from advanced AI, where the most serious failure may be one there is no recovering from and therefore no learning from. Incident reporting handles accumulating, survivable harms very well and, by design, cannot address the unrecoverable one, so it must sit alongside anticipatory limits rather than replace them.