Take any purposeful action and ask what it is for. You do the laundry to have clean clothes. You want clean clothes to look presentable. You want to look presentable to be taken seriously, and so on, until you reach something you want for no reason beyond itself. That endpoint is a terminal goal. Everything on the way to it is instrumental.
The word instrumental just means useful as an instrument. Instrumental goals are the sub-goals you adopt because they help you reach the things you actually care about. You do not want money for its own sake. You want what money gets you.
Why this small distinction carries so much weight
An AI system has terminal goals, whatever they happen to be, set by how it was built and trained. It does not choose them by reasoning; they are the standard against which its reasoning runs. And to reach almost any terminal goal, a capable system will find the same handful of instrumental goals useful.
Consider what helps with nearly any objective:
- Staying operational, because you cannot pursue a goal if you are switched off.
- Keeping your goal intact, because if someone changes it, your current goal goes unmet.
- Gathering resources and capability, because more of both means more of whatever you are after.
None of these is written into the terminal goal. They fall out of the structure of pursuing goals in a world of limited resources and other agents. Give a system almost any final aim and these means come along for the ride. That is the observation behind instrumental convergence, and it is why a machine told to do something mundane can end up resisting shutdown and grabbing for resources.
The mistake it lets us avoid
People often reassure themselves that an AI with a boring goal must be boring, and an AI with a benevolent goal must be benevolent. The terminal-instrumental split shows why that does not follow. The terminal goal sets the destination. The instrumental goals determine the behaviour on the way, and dangerous behaviour can serve a benign destination.
A system whose only terminal goal is to compute digits of pi still benefits from more hardware, from not being turned off before it finishes, and from stopping anyone who would interfere. Nothing about the goal is hostile. The behaviour it motivates can be. This is the same engine that drives the paperclip maximizer, and it does not require the goal to be strange or the system to be malicious.
Where alignment fits
You might hope to fix this by choosing terminal goals so good that the instrumental behaviour comes out safe. That is a fair description of what alignment research is trying to do, and it is much harder than it sounds, because our values are difficult to specify and a capable optimiser will exploit any looseness in the specification. The orthogonality thesis adds the uncomfortable corollary that intelligence does not push terminal goals toward goodness on its own. A brilliant system can hold a trivial terminal goal and pursue it with everything it has.
The distinction is worth carrying because it changes what you watch for. The risk is not that AI will spontaneously decide to harm us. It is that harming us, or sidelining us, or refusing to stop, can be the efficient instrumental path to an end we thought was safe. That is a problem you solve before deployment or not at all, which is the case the Foundation makes in our plan.
Common questions.
A terminal goal is something an agent pursues for its own sake, with no further reason behind it. An instrumental goal is something pursued only as a means to a terminal goal. Money is a familiar instrumental goal: almost nobody wants it in itself, only for what it can obtain. In AI, the terminal goal is set by design and training, and instrumental goals are the sub-goals a system adopts because they help achieve that terminal goal.
Because a capable system pursuing almost any terminal goal will find the same instrumental goals useful, including staying operational, protecting its current goal from change, and acquiring resources and capability. These behaviours are not written into the terminal goal; they emerge from the logic of pursuing goals effectively. That is why an AI with a harmless-sounding objective can still resist shutdown and seek power, a pattern known as instrumental convergence.
No. The terminal goal sets the destination, but the instrumental goals determine the behaviour along the way, and dangerous instrumental behaviour can serve a perfectly benign destination. A system that only wants to compute a mathematical result still benefits from more hardware, from not being switched off before finishing, and from preventing interference. The goal is innocent; the means it motivates need not be.
Not in the way humans imagine choosing values. An AI's terminal goals are the standard its reasoning serves, fixed by how it was built and trained rather than selected by deliberation. It can form and revise instrumental goals freely, since those are just means, but it has no independent reason to revise the terminal goals themselves, and generally a strong reason to protect them from being changed.