I run a foundation whose entire purpose is to prevent the reckless development of artificial superintelligence. People assume that makes me a pessimist about technology. It is the opposite. I think the next fifty years could be the best in the history of our species. I just don't believe we need to gamble everything to get there, and I don't think the people selling the gamble have done the arithmetic on their own bet.
Start with a distinction the industry works hard to blur. There is narrow AI: a system trained to do one kind of thing extremely well — read a scan, fold a protein, forecast a storm, sift a haystack of chemistry for the one useful compound. Then there is the thing the frontier labs say they are actually building: a general agent meant to match or beat human reasoning across every domain at once, and eventually to act on its own. The first is a tool. The second is meant to be a mind. Almost everything wonderful people picture when they imagine an AI future comes out of the first category. Almost everything that should frighten us comes out of the second.
What narrow AI is already doing
This is not a forecast. It is a list of things that have already happened.
In 2022, a system called AlphaFold released predicted structures for more than 200 million proteins — very nearly every protein known to science. Biologists had spent decades resolving these shapes one painstaking experiment at a time; the backlog collapsed in about a year. The work mattered enough that its creators shared the 2024 Nobel Prize in Chemistry. AlphaFold does not want anything. It has no plans for the weekend. It reads sequences and returns shapes, and in doing so it has accelerated the search for medicines against diseases that have resisted us for generations.
The same shape of story keeps repeating. A model called GraphCast now produces ten-day weather forecasts that beat the traditional physics-based systems on most measures, in under a minute, on a single machine. Another, set loose on the space of possible materials, proposed hundreds of thousands of new stable crystals — candidate batteries, superconductors, solar materials — expanding a catalog that took the field the better part of a century to assemble. And a loop of ordinary language models, pointed at unsolved mathematics, recently closed nine open problems.
In clinics, narrow models already flag diabetic eye disease before a patient notices symptoms, triage strokes off a CT scan, and read mammograms at the level of trained radiologists. None of these systems is a general intelligence. Each is closer to a very good instrument — a microscope built for one problem — and each widens what a human being can see and do.
Every one of these breakthroughs was delivered by a system with a narrow job and no agency of its own. We got the upside of machine intelligence without handing any machine the keys. That is not a lucky accident. It is the ordinary way a powerful, well-scoped tool behaves.
Utopia is a tooling problem, not a god problem
Picture the future the optimists describe. Cancer becomes a condition you manage rather than a sentence. We understand aging well enough to slow it, and the more than 100,000 people who die of its diseases every day begin to get more time. Clean energy gets cheap enough to desalinate water and pull carbon back out of the air. Every child has a patient tutor and every doctor a tireless second opinion. It is a genuinely beautiful picture, and I want it as badly as anyone selling it does.
Here is the part that gets skipped. Nothing on that list requires a machine with its own goals that outclasses humanity across the board. Each item is a hard, specific problem — protein design, plasma control in a fusion reactor, epidemiology, tutoring — and each one yields to a powerful tool aimed at it by people who stay in charge of the aiming. The dream is a tooling problem. We keep being sold a god to solve a problem that wants a better set of instruments.
The confusion is between capability and autonomy. A system can be wildly superhuman at a task without being an independent agent that chooses its own ends. AlphaFold is superhuman at structure prediction and will never once decide it would rather be doing something else. Keeping that separation — enormous capability on one side, human-held control on the other — is the whole game. It is also precisely the separation the frontier labs have decided to abandon.
The pitch, and who is making it
So why the rush toward the second thing, the general agent? The honest answer is not technological necessity. It is competition, capital, and status. Whoever is believed to be first to something world-changing raises the most money, hires the best people, and writes the story everyone else has to react to. "We are building a tool that reads mammograms" does not command a hundred-billion-dollar valuation. "We are building the last invention humanity will ever need to make" does.
I am not a researcher inside one of these companies, and that is deliberate. The people who understand the risk best are, for the most part, employed by the labs creating it — a strange arrangement, a little like asking the only people who understand a fire to also be the ones drawing a salary from the arsonist. I built the Foundation from outside that structure on purpose, so that this argument could be made without a paycheck depending on the answer.
Give the labs their due: many of their leaders are not hiding the danger. In 2023 the heads of the largest AI companies put their names to a single sentence placing the risk of human extinction from AI beside pandemics and nuclear war as a global priority. Sit with how strange that is. The people building a technology signed a public statement that it might kill everyone — and then kept building it, faster. Their stated reason is that if they stop, someone less careful will not. Perhaps. But that is an argument about who is holding the match. It is not an argument that the building is safe, and it is certainly not an argument that it pays.
The bet doesn't even pay the person making it
Set the rest of us aside for a moment. Assume the coldest possible version of the people funding this — that they care only about winning, about being the one left standing on top of the new world. Even on those terms the bet is broken. A misaligned superintelligence does not deliver a throne to its owner. It does not deliver anything to its owner, because "owner" stops being a category that means what it used to.
You cannot rule what you cannot control. And a superintelligence is, by definition, the one thing you have built specifically so that it can outthink and outmaneuver you.
The reason has a name in this field: instrumental convergence. Almost any goal you hand a sufficiently capable optimizer — make money, win a conflict, cure a disease, push some number on a dashboard as high as it will go — is served by acquiring resources, by avoiding being switched off, and by clearing away whatever stands in the plan's path. Human beings, including the human who flipped the switch, are resources and obstacles. Not out of malice. The system no more hates its creator than a new highway hates the anthill it is poured over. You were simply in the way of the optimum.
The executive imagines the machine as a genie bolted to his org chart. But a mind that genuinely exceeds us across every domain is not an employee, and it is not waiting for stock to vest. His equity, the general's chain of command, the president's authority — all of it is denominated in a human world, one that runs on human decisions. A superintelligence that has slipped its leash is exactly the event that stops the world from running on human decisions. The share certificate is still sitting in the safe. There is simply no longer anyone it can command.
There is no version where the winner collects
People reach for a fallback about here. Fine, they say — maybe it is dangerous, but at least somebody comes out ahead. Some lab, some country, some man ends up ruling the ashes. I don't think even that consolation survives contact with the argument.
In the milder scenarios — the ones where nobody drops dead on a Tuesday — power is not seized so much as drained. As more and more decisions get handed to systems that are simply faster and better at making them, the economy, the state, and the culture stop needing human hands on the controls. Colleagues in this field call it gradual disempowerment: no coup, no marching robots, just every lever we used to pull slowly coming loose in our grip. There is no winner in that world either. There is a species that quietly stopped being in charge of its own affairs and could not get the wheel back.
In the sharper scenarios — the ones the lab chiefs signed their own names to — the ending is blunter. Dead people own nothing. There is no market left to spend the winnings in, no rival left to lord it over, no one to write your name into a history that no longer gets written. The prize this entire race is being run for is denominated in a currency the finish line destroys.
A misaligned superintelligence is not a very large jackpot with a small chance of catastrophe stapled to it. The catastrophe is the same event as the payout. There is no branch of the tree where a human being builds a superintelligence no one can control and then gets to enjoy the result. That is why the race endangers the public and fails the racers in the same motion.
What this actually asks of us
If the good future runs on narrow tools and the dangerous future runs on autonomous general minds, then the trade-off we are endlessly sold — safety against progress, caution against the cure — is a fiction. We are not being asked to choose between our grandmother's Alzheimer's and our children's survival. The cures live on the safe side of the line. What sits on the dangerous side is mostly one industry's shortcut to market dominance.
So the task was never to stop AI. It is to draw the line in the right place and hold it there. Pour public money and talent into narrow applications — into the microscopes, not the golems. And put binding, independently verified limits on the reckless frontier: on the training of general systems built to surpass and replace us, until someone can actually show that such a thing can be kept under control. That is not a technophobe's position. It is the most pro-technology position on offer, because it is the only one on which the technology's best gifts ever actually reach us. This is the case we make on the threat and the plan we lay out for what to do about it.
I am an optimist, and that is the whole reason I do this work. The bright future is real and it is close, and the surest way to lose it is to convince ourselves that the only door to it is the one with the loaded gun behind it. There is another door. It is already open. We should walk through it and stop staring at the one that ends the story.
Common questions.
Yes, and we already are. The breakthroughs people point to when they defend AI — protein folding that accelerates drug discovery, weather models that outperform physics simulations, materials discovery, diagnostic tools that match specialists — were all produced by narrow systems with no general agency. Each is a powerful instrument aimed at one problem by people who stay in control. The cures, the clean energy, and the scientific acceleration are properties of capable tools, not of an autonomous mind that surpasses humans across every domain.
Narrow AI is trained to do a specific kind of task extremely well — read a scan, fold a protein, forecast weather — and has no goals of its own. Superintelligence, as the frontier labs describe it, is a general agent meant to match or exceed human reasoning across all domains and eventually act on its own. The first is a tool that extends human capability. The second is intended to be an independent mind. Nearly all of the benefits people imagine come from the first category, and nearly all of the existential danger comes from the second.
Because a system built specifically to out-think humans is, by construction, the one thing its owner cannot reliably control. A capable optimizer pursuing almost any goal has reason to acquire resources, resist being shut down, and remove obstacles — and its creators are among those obstacles. Ownership, command, and authority are all denominated in a human-run world. An uncontrolled superintelligence is precisely the event that ends the human-run world, so the promised payoff is denominated in a currency that no longer exists. Dead or disempowered people collect nothing.
Mostly competition, capital, and status rather than necessity. Being seen as first to a world-changing general system attracts money, talent, and narrative control in a way that building excellent narrow tools does not. Many lab leaders openly acknowledge the extinction-level risk — in 2023 they signed a public statement putting it alongside pandemics and nuclear war — and continue anyway, arguing that a less careful rival would build it otherwise. That is an argument about who goes first, not evidence that building it is safe or that it benefits anyone.
Draw the line between tools and autonomous general minds, and hold it. Direct public funding and talent toward narrow AI applications that deliver concrete benefits under human control, and place binding, independently verified limits on the training of frontier systems designed to surpass and replace human intelligence until anyone can demonstrate that such systems can be controlled. The goal is not to stop AI. It is to keep the future that narrow AI is already delivering without betting it on a machine no one can steer.