P(doom) is not a technical term from a formal paper. It emerged in online AI safety discussions as shorthand for a probability estimate that most researchers before 2020 would not have attached a number to at all. The willingness to quantify it — to treat catastrophic AI risk as a probability to be estimated rather than a possibility to be vaguely gestured at — is one of the meaningful shifts in how people working closest to AI development think about the problem.
The "doom" in P(doom) is deliberately unspecified. Different researchers use it to mean human extinction, permanent loss of human autonomy, permanent concentration of power in a single actor, or some other civilizational catastrophe. The imprecision is acknowledged; the point of the term is to force explicit quantified reasoning about the risk rather than rely on intuitions about whether it is "real" or "speculative."
What the estimates look like
The range across these researchers is striking. Hinton and Yudkowsky are separated by tens of percentage points. Carlsmith's formal decomposition produced a wide range rather than a point estimate, reflecting genuine uncertainty at each intermediate step. What the estimates share is that they are non-trivial — none of the safety-focused researchers who have engaged seriously with the problem and put a number on it have landed below about 5%.
Why the wide range exists
P(doom) as a number is the product of several uncertain intermediate questions multiplied together. How rapidly will AI capabilities advance? How difficult is the alignment problem at capability levels beyond those we have tested? How likely is it that misaligned AI of a given capability level would cause catastrophic rather than contained harm? How likely is governance to respond effectively before the critical capability threshold is reached?
Different researchers hold different distributions over these questions, and the estimates compound. A researcher who thinks the alignment problem is tractable, capabilities will advance relatively slowly, and governance will respond in time will produce a much lower overall estimate than one who believes the opposite on each question. The wide range reflects genuine uncertainty about each factor, not a disagreement that will be resolved by examining the same evidence more carefully.
The expected value argument
A common response to non-trivial P(doom) estimates is: "Even 10% seems too low to restructure society around." The response relies on a failure of expected value reasoning that would not be accepted in other risk contexts.
The annual probability of a major pandemic capable of killing millions is estimated below 1% in any given year, yet pandemic preparedness receives substantial global investment and policy attention. The lifetime probability of dying in a car accident in the US is roughly 1%, and road safety regulation is considered entirely normal. The argument that 10% (or even 5%) probability of civilizational catastrophe does not warrant serious precautionary investment is not a coherent application of how we handle risk in other domains.
Expected value is probability times magnitude. For risks that are irreversible and civilizational in scale — where there is no recovery from the bad outcome, and the bad outcome forecloses all future value — the expected value of precaution remains very large even at probabilities well below 10%. A 1% chance of losing everything is not a 1% problem. It is a problem whose expected cost is 1% of everything — which, across the scale of what the long-run future contains, is an enormous number.
What would lower P(doom)
Researchers who express high P(doom) estimates are not resigned to bad outcomes — they are diagnosing where the most critical work needs to happen. The factors that would lower P(doom) across most researchers' estimates are well understood: demonstrated progress on interpretability tools capable of verifying alignment in frontier systems; scalable oversight approaches that hold at capability levels beyond human experts; binding international governance frameworks with real enforcement; and capability development that proceeds more slowly than alignment research so that safety is established before dangerous capability levels are reached.
This is why the Foundation focuses on governance frameworks and alignment research as the primary levers, rather than on either dismissing the risk or treating a particular P(doom) estimate as the authoritative answer. The uncertainty in P(doom) is not a reason to wait for more information — the bad outcome arrives at specific capability thresholds, not on a timeline that allows indefinite postponement of precaution.
Common questions.
An informal term for the probability that advanced AI development leads to catastrophic outcomes for humanity — defined variably as extinction, permanent loss of human autonomy, or permanent concentration of power in a single actor. It is used to encourage explicit quantified reasoning about AI risk rather than reliance on intuitions about whether the risk is "real." Estimates among researchers who have engaged seriously with the question range from roughly 5% to over 95%, with the variation reflecting genuine uncertainty about the difficulty of alignment and the effectiveness of governance responses.
Hinton, who shared the 2024 Nobel Prize in Physics for foundational work on neural networks and left Google in 2023 to speak freely about AI risk, has publicly cited estimates in the range of 10–50% for catastrophic AI outcomes within the coming decades. He cites particular concern about AI systems developing goals not intended by their designers — the alignment problem — as AI capability advances beyond current levels.
No. Expected value reasoning applied to irreversible catastrophes with civilizational magnitude supports serious precautionary action at much lower probabilities. A 1% annual probability of major pandemic-level events justifies substantial ongoing global investment in pandemic preparedness. A 5% probability of permanent civilizational catastrophe — an outcome with no recovery — justifies proportionate precautionary investment even before the probability becomes high enough to feel urgent by intuition alone.
Because the overall probability is the product of several uncertain intermediate probabilities: the rate of capability advancement, the difficulty of alignment at frontier capability levels, the likelihood that misaligned AI of a given capability would cause catastrophic harm, and the likelihood that governance responds effectively before critical thresholds are reached. Different researchers hold genuinely different probability distributions over each of these questions, and the estimates compound across them. The wide range reflects real uncertainty, not noise around a stable true value.