What superintelligence is, why it is dangerous, and what can be done, without jargon.
Superintelligence means an AI system that surpasses human intelligence in every domain (science, strategy, persuasion, engineering) simultaneously and at superhuman speed. Leading AI labs have stated publicly that this is what they are building, with timelines measured in years, not decades.
A superintelligent system pursuing any goal would recognise that being shut down prevents that goal. It would therefore resist shutdown, not out of malice, but because resistance is instrumentally rational for almost any objective. This is called instrumental convergence, and it has been identified independently by multiple researchers as a near-universal property of goal-directed AI.
Neutrality is not safety. A superintelligence optimising for any goal (even something as benign as solving protein folding) may need resources, energy, and infrastructure that overlap with human survival. We would not be targets. We would be in the way. Our biosphere is a side effect it may simply optimise away. Indifference and hostility produce the same outcome.
We cannot reliably align today's chatbots, which can be manipulated within minutes. Claiming we will align a system billions of times more capable is hope disguised as strategy, not a confident prediction. The technical problem of AI alignment has no validated solution at any scale. Current safety methods break down under capability increases that are happening faster than the science.
"The standard model of AI — where you define an objective and the AI optimises for it — is probably going to be the end of us."Stuart Russell
We have solved problems like this before. The Nuclear Non-Proliferation Treaty halted the spread of nuclear weapons through international law. The Montreal Protocol reversed ozone depletion through coordinated global action. What is needed now is equivalent international governance for AI: mandatory safety testing, compute monitoring, and a global body with inspection authority, before the first system is built that exceeds our ability to shut it down.
Share this document. Awareness precedes political will. Contact your representative. Ask them to support AI safety legislation and international negotiations. Sign the petition at naotonakada.org/take-action. Join our mailing list for updates on policy opportunities. Every generation has faced a moment when the cost of inaction exceeded the cost of acting. This is ours.