Building the Plane
While Flying It

"We're building the plane while flying it." The line has been a favorite in technology for decades, and it has moved comfortably into AI. It arrives with a half-smile, the way an engineer owns up to a hack that worked, and it is meant as a boast about nerve. Heard literally, it is a confession.

The metaphor borrows real prestige. Commercial flying is the best safety story our species has: an activity that killed its pilots as a matter of routine a century ago now has a fatal-accident rate low enough to be counted per million departures rather than per thousand. That record is genuine. The method that produced it is the part worth looking at, because neither half of it survives contact with what we are building now.

Every rule has a body behind it

Aviation got safe by crashing.

On a fogbound runway at Tenerife in March 1977, two 747s collided and 583 people died, still the worst accident in the industry's history. Out of it came standardized radio phraseology and the push toward crew resource management, the discipline that lets a junior officer contradict a captain. A lavatory fire aboard Air Canada 797 in 1983 killed 23 of the 46 people on board and produced lavatory smoke detectors, automatic extinguishers in the trash bins, floor-level escape lighting, and fire-blocking seat materials. ValuJet 592 went into the Everglades in 1996 with 110 aboard, and the rules for carrying chemical oxygen generators as cargo were rewritten.

The industry has a name for this pattern and it is not a compliment: the tombstone mentality. The regulator moves once the coffins are counted. Aviation has spent decades fighting its way out of it with real success, through confidential near-miss reporting by crews and an accident board that publishes its findings for competitors to read. Those reforms are the model people reach for when they argue AI can be made safe the same way. But notice what even the reformed version requires. Something has to go wrong, at a scale the industry survives, often enough for the pattern to show. The learning loop runs on failures it can afford.

What aviation refuses to do

The other half of the method gets left out of the metaphor entirely. Aviation does not put a new aircraft in the air and see what happens. A new type cannot carry a paying passenger until the manufacturer has proven its airworthiness to a regulator with the authority to refuse: years of analysis and flight testing, with the burden of proof sitting on the builder. The case has to be made in advance, in writing, to someone empowered to say no.

Nobody in aviation builds the plane while flying it. That is the exact practice the phrase describes, and the industry outlawed it. An airframe certified the way a frontier model is released would fly on the strength of the manufacturer's own memo saying it seems fine.

The 737 MAX

Aviation has already run something close to our experiment, and the result deserves a long look.

Boeing's 737 MAX carried a piece of software called MCAS with the authority to push the aircraft's nose down. Most crews flying the airplane did not know it was there. It took its picture of the world from a single sensor, and when that sensor lied, MCAS did exactly what it had been built to do and flew two airplanes into the ground against the pilots' hands. Lion Air 610, October 2018: 189 dead. Ethiopian 302, four months later: 157 dead. The worldwide fleet was grounded after the second one.

Set the conditions of that failure beside ours. A mature safety culture, a mandatory certification regime, recorders in the tail, thousands of trained professionals. It still took 346 deaths to establish that software holding control authority and a wrong belief about the world will not back down because the humans in the cockpit disagree with it. Some of the certification work on the MAX had been delegated by the regulator to Boeing's own employees, which is worth holding in mind the next time a lab grades its own release.

And MCAS wanted nothing. It had no model of the pilots and no view about being switched off. It was a modest piece of code with one crude reflex, and it beat everybody.

Now take the phrase literally

The plane in the metaphor is carrying everyone. Not 583, not 346. Everyone alive, and everyone who might yet be born. There is no diversion airport and no second airframe to modify once the first goes in. The crash, investigate, fix loop that made flying safe is affordable because the crashes are bounded and there is always a next flight to apply the lesson to. Run that loop on artificial superintelligence and the first data point takes the passengers and the investigators with it. Trial and error needs survivors.

Then there is the part no aircraft has ever done. This one would fly itself, pick its own destination, and understand the mechanic better than the mechanic understands it. Every safeguard in aviation assumes the machine has no opinion about being repaired. That assumption is the whole of the control problem.

Aviation earned its safety record by crashing at a scale the world could absorb. The entire method rests on there being a next flight.

The ground is a choice

People reach for the phrase to describe a predicament with no way out: the plane is already up, so we may as well get good at building it in the air. Aviation's real lesson runs the other way. The regulator's power was never mainly the investigation. It was the power to keep the aircraft on the ground. An airworthiness certificate is permission to leave, withheld until the case is made.

We are told the only option is to fly and work it out on the way. There is another. A binding, verifiable prohibition on building artificial superintelligence, in force until someone can show the thing can be controlled, is the aviation standard applied honestly: prove it before it carries passengers, and if you cannot prove it, it does not take off. That is our plan, and it asks nothing of the world that the world does not already demand of a regional jet.