The Five Biggest Myths About Governing Superintelligence

The objections to governing superintelligence do something clever. They concede the danger — "sure, maybe it's risky" — and then quietly relocate the impossibility from the technology to the solution. If the risk can't be denied, perhaps the response can be made to look naive, or dangerous in its own right, or simply hopeless. The effect is the same as denying the risk outright: nothing gets done.

What follows are the five I hear most, stated as strongly as I can make them, because a rebuttal to a weak version convinces no one. A related family of objections — that China won't comply, that companies will regulate themselves, that we can just switch a bad system off — I've answered on the Myths page. These five are about the governance itself.

"Governing AI globally means world government — and tyranny."

The fear here is honorable, and I don't want to caricature it. Concentrate enough authority in one body to control the most powerful technology on Earth, and you may have built the machinery of exactly the tyranny you feared from the technology. People who distrust centralized power are right to distrust it. The instinct is sound.

But it is aimed at a straw version of the proposal. What is actually on the table is not a global authority over daily life, or over software in general, or over the phone in your pocket. It is a narrow regime pointed at one specific thing: the small number of extraordinarily large training runs capable of producing a system more powerful than any human across the board. The model is the nuclear non-proliferation regime or the Chemical Weapons Convention — targeted agreements that watch fissile material and precursor chemicals, not world governments that dissolved the nation-state. Nobody thinks the International Atomic Energy Agency runs the planet. It monitors a specific, dangerous activity and leaves everything else alone. That is the template.

And here is the turn that the objection misses entirely. The surest path to permanent tyranny is not the treaty. It is the absence of one. An uncontrolled superintelligence, or one captured by a single company or state that solves control before anyone else, is the most complete instrument of domination ever conceivable — a power that never dies, never tires, and can lock in one set of values forever, past any hope of revolt or reform. Every tyranny in history has ended because tyrants are mortal and their machinery is human. This one need not be. If you are frightened of unaccountable power — and you should be — an enforceable agreement keeping any single actor from seizing that power is the anti-authoritarian position, not the authoritarian one. Democratic oversight is the whole point.

"You can't verify it — it's just software on a server somewhere."

This is the most technically serious objection, and it sounds decisive. Nuclear treaties work, the argument goes, because you can count warheads and detect enrichment; uranium is physical and rare. But a model is just numbers. You can't inspect for a file. Any agreement would be unenforceable theater, binding the honest and doing nothing about the cheat.

It sounds decisive until you notice you're verifying the wrong thing. You don't police the weights; you police the compute that produces them — and compute is about as physical and rare as uranium. Training a frontier system takes tens of thousands of specialized chips running for months in a single facility, and that supply chain is one of the narrowest chokepoints in the modern economy: essentially one Dutch company makes the lithography machines, one Taiwanese company fabricates the leading chips, one American company designs the dominant architecture. The clusters that use them draw so much power they show up in electricity data and in satellite imagery. You can count the chips, trace their sale, meter the power, and watch the buildings. Fissile material is governed by exactly this logic — track the scarce physical input and the weapon can't be built in secret. The verification playbook already exists; the inputs here happen to be even more concentrated. It is an engineering problem, and engineering problems have solutions.

"Regulation will smother innovation and let the reckless win."

The strong form: technology advances because someone is free to build, most regulation is written by people who don't understand what they're slowing, and a heavy hand on AI would kick the most important industry of the century to whoever regulates least. Better a flawed frontier here than a pristine one somewhere hostile.

Start with a distinction the objection erases. Nobody serious is proposing to regulate "AI." The beneficial systems folding proteins, reading scans, tutoring children, and modeling the climate are narrow, controllable tools, and a superintelligence agreement leaves them entirely alone — that bright, ordinary future is the one we're trying to protect. The target is a single hazardous class: training runs reaching for general capability beyond ours that no one yet knows how to make safe. Restraining that is no more "anti-innovation" than gain-of-function rules are anti-biology.

Then the harder point, the one the race-logic never confronts. Going fast does not deliver a safe superintelligence sooner. It delivers an unsafe one, and an unsafe superintelligence does not serve the country or company that built it — it threatens them along with everyone else. When corners get cut under competitive pressure, the corner that gets cut is the one that would have made the system controllable. So the "innovation" being defended here is innovation whose likeliest endpoint is a system its own makers can't steer. That isn't progress that governance would spoil. It's the hazard itself, wearing progress as a costume.

"It's premature — you can't regulate something that doesn't exist."

This feels like maturity. Don't legislate against phantoms; wait until the technology is real, understand what you're dealing with, then write rules fitted to the facts instead of to speculation. Regulating hypotheticals is how you get bad law.

It would be wisdom for almost any other technology. It is a trap for this one, and the reason is timing. Governance is slow: the Nuclear Non-Proliferation Treaty took years to negotiate and the Montreal Protocol took years more to bite, and those were the fast ones. If you wait until a superhuman system plainly exists to start building the institutions to govern it, the institutions arrive years after the thing they were meant to contain — and this is the one technology whose first serious failure may leave no one to pass the second law. "Wait for proof" is a sound rule when the proof is survivable. Here the proof is the catastrophe. The precautionary logic isn't timidity; it's the recognition that the lead time for the cure is longer than the fuse on the problem.

Every workable safeguard for a catastrophic technology was built before or during its rise — never after, because "after" was never on offer.

"Democracies are too slow and captured — it'll never pass."

The most quietly corrosive of the five, because it wears the coat of realism. Look at the gridlock, the lobbying, the short attention of legislatures, the reach of the industry's money. Even if a treaty were wise, the machinery to produce it is broken. Better to be clear-eyed than to waste a decade on a fantasy.

I understand the exhaustion behind this. But it is refuted by the historical record, repeatedly, and usually against worse odds than we face now. The idea that rival powers would sign away their deadliest options was called hopelessly naive — right up until they did, in the Non-Proliferation Treaty, the Biological and Chemical Weapons Conventions, the Montreal Protocol, the Antarctic Treaty. Adversaries who were fighting proxy wars and calling each other evil still built verified arms control at the height of the Cold War. Each of those looked impossible from the near side and inevitable only in hindsight.

And the ground is more favorable than the cynicism admits. Across the United States, the United Kingdom, and the EU, large majorities already tell pollsters they want AI developed cautiously and bound by strong rules; the public is not the obstacle. The obstacle is that this widespread, quiet concern has not yet been organized into pressure a legislature can feel. That is not a wall. It is a job — the specific, doable job of turning latent agreement into political will. Which is precisely the work in front of us.

What the five have in common

Set them side by side and the pattern is plain. Each objection grants the danger and then finds a different place to give up — the solution is too big, or unverifiable, or too costly, or too early, or too hard to pass. Any one of them, taken alone, sounds like prudence. Together they form a machine whose only output is inaction, and inaction is not neutral here. It is a choice to let the most consequential technology in history be built with no binding limits at all, by a handful of competing firms, under the exact pressure most likely to produce the outcome everyone claims to fear.

None of these is answered by insisting governance will be easy. It won't. Verification will be contested, negotiations will be slow, and some governments will drag their feet — every hard-won treaty in history had all of that and got built anyway. The honest claim is not that a superintelligence agreement is simple. It is that it is possible, that the pieces to build it already exist, and that the reasons for waving it away collapse the moment they are examined instead of repeated.

That is the whole wager of this Foundation. Not that the work is easy — that it is worth doing, and that "it can't be done" has been wrong about every comparable effort we have ever made. The task is to build the political will for a binding, verified agreement that keeps the most dangerous systems from being built until they can be made safe. The objections are real. "Therefore give up" has never been the answer, and it is not the answer now.