The precautionary principle emerged from environmental law and is now embedded in international instruments. Its best-known formulation is Principle 15 of the 1992 Rio Declaration: where there are threats of serious or irreversible damage, lack of full scientific certainty shall not be used as a reason for postponing cost-effective measures to prevent it. Variants appear throughout European Union law and in treaties on climate, biodiversity, and hazardous substances. The principle exists precisely for situations where waiting for proof means waiting too long.
Why it fits AI so well
The case for AI risk has an awkward feature: the most serious scenarios have not happened, and cannot be demonstrated in advance without running the experiment. Critics use this to dismiss the concern — no proof, no problem. The precautionary principle is the direct answer. It was built for threats that are grave and potentially irreversible but not yet certain, and it inverts the burden: the question is not whether catastrophe is proven, but whether the possibility is serious enough to warrant protective action given what is at stake.
Two features of frontier AI make it close to a textbook case for the principle. The potential harm is on the highest end of the severity scale — plausibly catastrophic and irreversible. And the science is genuinely uncertain, with credible experts assigning meaningful probability to loss-of-control scenarios and no reliable method to rule them out. Grave, irreversible, uncertain: this is exactly the profile the precautionary principle was written to address.
What precaution requires in practice
- Shifting the burden of proof. Developers of the most capable systems should bear the burden of demonstrating safety before deployment, rather than society bearing the burden of proving danger after the fact — the model already used for pharmaceuticals and nuclear plants.
- Acting before certainty. Governance should not wait for a demonstrated catastrophe to justify binding limits, because for irreversible risks the demonstration is the disaster.
- Proportionality to the stakes. The scale of precaution should match the scale of potential harm; the highest-severity risks justify the strongest measures.
- Reversibility as a priority. Where possible, keep options open and avoid steps that foreclose future correction — a principle directly relevant to a technology that could entrench itself.
The objections, taken seriously
The precautionary principle has real critics, and honest advocacy engages them. The strongest objection is that, taken to an extreme, precaution can paralyse: almost any activity poses some conceivable catastrophic risk, and a principle that says 'act against all unproven grave threats' could justify blocking beneficial innovation on speculative grounds. Critics also note that precaution has its own costs — forgone benefits, including the possibility that AI itself could avert other catastrophes — and that these belong in the calculation.
These points do not defeat the principle; they discipline its use. The answer is that precaution is not 'ban everything that could conceivably go wrong'. It is proportional action against threats that are both serious and credibly supported, weighing the costs of action against the costs of inaction. Frontier AI qualifies not because someone can imagine a bad outcome, but because credible experts assign real probability to catastrophic, irreversible loss of control. The principle asks for measures proportionate to that specific, well-supported risk — not for a blanket halt to technology.
For an irreversible risk, 'wait for proof' is not caution — it is the opposite. The precautionary principle exists so that we are not required to suffer the catastrophe in order to earn the right to prevent it.
Naoto Nakada, Founder · Nakada Foundation to Save Humanity
The foundation for acting now
The precautionary principle matters for AI governance because it dissolves the most common argument for delay — that we should not regulate until the risks are proven. In law and in ethics, that argument fails for exactly the class of threat AI represents. Grave, irreversible, and uncertain harms are the paradigm case for acting before certainty, with the burden on those creating the risk to show it is contained. This does not settle what the measures should be; that is the work of treaty design, verification, and thresholds. But it settles the prior question of whether it is legitimate to act at all under uncertainty. It is — and the principle explains why.
Common questions.
A principle in international and environmental law holding that where an activity threatens serious or irreversible harm, a lack of full scientific certainty should not be used as a reason to postpone protective action. Its best-known statement is Principle 15 of the 1992 Rio Declaration, and it appears throughout EU law and treaties on climate, biodiversity, and hazardous substances. It exists for situations where waiting for proof means acting too late.
Frontier AI fits the principle's core case almost exactly: the potential harm is catastrophic and potentially irreversible, and the science is genuinely uncertain, with credible experts assigning real probability to loss-of-control scenarios that cannot be ruled out. The principle answers the common objection that AI risk is unproven by shifting the question from 'is catastrophe proven?' to 'is the possibility serious enough to warrant protective action given the stakes?'
It would shift the burden of proof onto developers of the most capable systems to demonstrate safety before deployment, rather than requiring society to prove danger after the fact — as already happens with drugs and nuclear plants. It would justify binding limits before a demonstrated catastrophe, call for measures proportionate to the severity of the risk, and prioritise keeping future options open against a technology that could entrench itself.
The strongest is that, taken to an extreme, it could paralyse beneficial innovation, since almost any activity poses some conceivable catastrophic risk, and that precaution has its own costs, including forgone benefits. The answer is that the principle calls for proportional action against threats that are both serious and credibly supported — not a blanket ban. Frontier AI qualifies because credible experts assign real probability to catastrophic, irreversible outcomes, not merely because harm is imaginable.