The phrase comes from weather. Forecasters use it for the uncommon case when several independent systems line up and feed the same patch of ocean — a low-pressure trough, a dying hurricane throwing off moisture, a ridge of high pressure that pins the whole thing in place. None of the ingredients is remarkable by itself. Sebastian Junger borrowed the term for his 1997 book about the 1991 storm that formed off the northeast coast and took down the Andrea Gail with all six hands aboard. What made it famous was not size. It was coincidence: routine dangers arriving together, producing something none of them held alone.
We tend to argue about artificial superintelligence as though the danger rests on one question. Will it be conscious, will it turn hostile, will it even get that capable — pick whichever crux you like, and the discussion settles into two camps trading intuitions. That framing is a mistake. The risk does not come from any single property of a superintelligent system. It comes from the way a handful of separately manageable problems stop being manageable the moment they occur at once.
A storm is just a coincidence of hazards
Here are the conditions that hold for superintelligence. Read them the way they are usually discussed: as a list of distinct engineering problems, each with its own research program and its own optimists.
Taken one at a time, none of these is unprecedented. Number four is the old problem of instrumental convergence, and number three is the intelligence explosion that researchers have modeled for years. Number five is why a system can look aligned in the lab and behave differently in the world, a failure mode with its own name: deceptive alignment. Number six is the race that every frontier lab is already caught inside. Each has been studied. Each has people working on it who believe it can be handled.
Take any one away and it looks survivable
Each condition, on its own, is the kind of thing engineers deal with all the time. We routinely build systems we do not fully understand and make them safe by letting them fail in small ways first. We manage products that cannot easily be recalled by testing them hard before release. We handle competitive races with regulation. What we have never had to handle is all of them at once, with each one quietly disabling the fix for the others.
If we could recall it, an accident would be recoverable. If we could test it honestly, we could catch the failures before they shipped. If nobody were racing, we could take the decades a decision like this obviously deserves. Every one of those escape routes exists for some technology or other. For this one they close at the same time. That is the whole of the perfect-storm claim — not that superintelligence is uniquely powerful, but that it is the first case where every exit shuts at once.
The method that made everything else safe
There is really only one technique humanity has ever used to make a dangerous technology safe, and it is not foresight. It is failure, studied closely.
The first jet airliner, the de Havilland Comet, began coming apart in mid-air in 1954. Investigators raised the wreckage from the Mediterranean, traced the cracks to metal fatigue spreading from the corners of its square windows, and the industry learned to round its windows and test for fatigue. Cars earned seatbelts, crumple zones, and airbags across decades of counting the dead on the roads. Nuclear power wrote most of its safety regime after Three Mile Island in 1979 and Chernobyl in 1986. The Food and Drug Administration got the power to keep unsafe products off the market after a sulfa medicine mixed in a solvent used in antifreeze killed more than a hundred people in 1937, and got the power to demand proof of safety before sale after thalidomide deformed thousands of infants at the start of the 1960s.
The pattern never changes. Build it, watch it hurt someone, take the accident apart, correct the design. It is an excellent method, and civilization owes most of its safety to it. It also rests entirely on two conditions: that the failure is survivable, and that somebody is left afterward to apply the lesson. Superintelligence is the case that may honor neither. A serious failure of a system more capable than us, spreading faster than we can respond and impossible to recall, is precisely the sort of failure you do not get to learn from. You cannot run the loop when the first bad pass through it ends the loop.
Trial and error is not one option among many for managing risk. It is the option — the mechanism underneath crash testing, clinical trials, building codes, and every other safety institution we have. All of it assumes you can afford the first failure and study the wreckage.
Superintelligence is the technology built to break that assumption. It removes the ability to iterate at the exact moment iteration is the only tool that has ever worked. What replaces it is a single attempt, run live, with no draft and no do-over.
Why the odds don't rescue the bet
The usual reply to all of this is that it assumes the worst. Catastrophe is not certain, the objection runs, so the alarm is overdone. The premise is right. Nobody serious claims certainty. The conclusion does not follow from it.
Russian roulette is not a prediction that you will die. Six chambers, one round, and the odds sit firmly in your favor — you will almost certainly stand up and walk away. People decline to play anyway, and their reasoning is not statistical. It is that the reward waiting on the other side of the trigger is not worth any chance at all of an ending you cannot come back from. The number is beside the point. The irreversibility is the point.
And the number here is worse than one in six. The disagreement among the people who understand these systems best is not a disagreement between safe and dangerous. It runs from something like one in ten to something like one in three, and the estimates tend to rise, not fall, the closer the expert sits to the frontier. Geoffrey Hinton, who did as much as anyone to build the field and left Google in 2023 to speak plainly about its risks, has put the chance that AI leads to human extinction within thirty years at ten to twenty percent. He is not the most alarmed voice in the field. Others place the figure higher.
Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.
Statement on AI Risk, Center for AI Safety (2023)
That sentence was signed in 2023 by the chief executives of the leading AI labs and by hundreds of the most cited researchers alive. Read it once more. The people building the technology placed its potential to end humanity in the same breath as nuclear war, and then went back to building it. That is the situation, described by its own participants.
Now set the reward beside the risk. You would not board a plane sold to you as one chance in a thousand of going down. The field's own estimate for superintelligence is a hundred times worse than that, and the aircraft is not a plane. It carries everyone alive and everyone who might yet be born. Against a downside that size, no upside survives being multiplied by even a modest probability of losing all of it.
The promised upside is real, and worth wanting — cures for diseases that have hunted us for all of history, an end to scarcity, problems solved that we have never been able to solve. But it keeps. A cure discovered in 2055 rather than 2035 is a tragedy measured in lives, and it is a tragedy we could still recover from and mourn. Extinction is the one result with no later attempt. The prize can wait for us to get this right. The loss cannot be taken back.
The objections worth taking seriously
Three counterarguments deserve better than a wave of the hand.
The first is that this is speculation — that we are nowhere near systems like this, and that dressing a distant maybe up as an emergency helps no one. Timelines are genuinely uncertain, and anyone who hands you a confident date is guessing. But the shape of the argument does not depend on a date. If the first serious failure cannot be undone, the safeguards have to exist before you arrive, not after, which means the work begins while the threat still looks far off. And the forecasts have been wrong mostly in one direction. Capabilities that were supposed to be decades away have kept turning up years early. Betting on plenty of time is betting against the recent record.
The second is that restraint is naive — that if we slow down, someone reckless in another company or another country simply builds it first. This is real, and it is the sixth condition on the list rather than a rebuttal to it. A race whose finish line might be a cliff is not won by pedaling harder. It is survived by getting everyone to agree on where the edge is. That points toward a binding, verifiable agreement among all the serious players, which is hard, rather than toward a sprint, which is merely fatal if the fear behind it is correct. The people who insist a treaty is impossible are usually the ones who would have to slow down under it.
The third is the most reasonable of the three: that alignment will probably be solved by the time it matters. Perhaps it will. But hold on to the word probably. We do not certify a bridge or a reactor on the expectation that the safety case will probably come together before the load arrives. For a one-shot, irreversible technology, probably is not a standard any other field would accept — and it is strange that this is the one place we are asked to accept it.
A decision too big to leave to the people making it
Put the pieces back together. A choice that is irreversible, one-shot, and civilization-wide, made under deep uncertainty, by a small number of competing firms with enormous financial reasons to make it quickly. Stated that way, the governance question nearly answers itself. This is exactly the category of decision that societies take out of the hands of the parties who stand to gain most from speed.
We have done it before, and every time over the same objections. Nuclear testing was pulled under international limits despite the commanders who wanted a free hand. Whole classes of weapons were signed away in the Biological and Chemical Weapons Conventions. The chemicals eating the ozone layer were phased out by the Montreal Protocol, over the protests of the companies that manufactured them. None of it happened because the people creating the hazard volunteered to stop. It happened because everyone else decided the risk was not theirs to run alone.
That is the entire reason this Foundation exists, and it is worth being exact about the claim. Not that catastrophe is certain — it is not. The claim is that this gamble is one no company and no country has the standing to make on behalf of everyone alive, and that the only instrument built to the scale of a risk this size is binding international law: verified, enforced, and in place before the storm rather than after it.
We get one attempt. Everything about how seriously we treat this should follow from that single fact.
Common questions.
Because the danger does not come from any single property of the technology. It comes from several ordinarily manageable problems occurring together: a system more capable than the people overseeing it, one that cannot be recalled once it spreads, that may improve itself faster than we can react, that develops dangerous subgoals from almost any objective, that we have no reliable way to test, and that is being built in a competitive race which punishes caution. Each of these is survivable on its own. Arriving at the same moment, each one disables the safeguard that might have caught the others.
Every dangerous technology we have made safe — aircraft, cars, nuclear power, medicine — became safe through trial and error. It failed, people were harmed, we studied the failure and corrected the design. That method depends on two things being true: that the failure is survivable, and that someone remains to apply the lesson. A superintelligence that fails catastrophically, spreading faster than we can respond and impossible to recall, may satisfy neither. The first serious accident could remove the chance to learn from it.
Catastrophe is not certain, and nobody serious claims it is — that is the point of the comparison. Russian roulette is not a prediction that you will lose; the odds favor you. People decline anyway, because no reward justifies a small chance of an outcome you cannot come back from. Researchers closest to the technology commonly estimate the risk of catastrophe between one in ten and one in three. You would not board a plane with a fraction of that failure rate, and the stakes here are not a single plane. See our explainer on P(doom) for where those numbers come from.
Fixing problems as they arise is the trial-and-error method, and it needs failures you can survive and undo. This technology may allow neither. "Alignment will probably be solved in time" is a hope, not a plan. No other high-consequence field — aviation, nuclear power, medicine — is allowed to run a one-shot, irreversible system on the assumption that the safety case will probably come together before it is needed.
Binding international governance, independently verified, and in place before the most advanced systems are built rather than after. The precedents exist: limits on nuclear testing, the Biological and Chemical Weapons Conventions, and the Montreal Protocol. None arrived because the parties creating the hazard volunteered to stop. Each came from outside pressure and enforceable law. Applying the same logic to superintelligence is the work the Nakada Foundation exists to do.