This is Part 6 of Whatever You Ask For, a series on machines that grant wishes exactly as worded, and the wording that remains ours.
Asking It Backwards
Everything so far has assumed that a person writes the objective. A human being decides what counts as success, hands it to a system, and lives with the consequences of having written a finite sentence around an infinite space. The obvious question, and one that a serious research community has spent a quarter of a century on, is whether that step can be automated too. Can a machine work out what we want, so that we do not have to spell it out?
The most elegant version of the idea reverses the usual direction of the problem. Ordinarily you are given an objective and you search for the behaviour that best satisfies it. Stuart Russell, in a short conference paper in 1998 and then in a 2000 paper with Andrew Ng that gave the field its name, asked what happens if you run the arrow the other way. Suppose you are given the behaviour, and you search for the objective that would make it optimal. This is inverse reinforcement learning, and its appeal is immediate. It seems to dissolve the entire problem of this book. You would not have to write down what you want. You would only have to act, and be watched.
The appeal is not naive. It is how humans transmit most of what cannot be said. An apprentice watches a master and absorbs a thousand judgments the master could never articulate. A child learns what a household values less from what it is told than from what it sees rewarded and ignored. We infer goals from behaviour constantly, effortlessly, and mostly correctly, in exactly the way the idea proposes a machine might. If people can do it, the reasoning goes, the process must be learnable.
Watch a joiner teach an apprentice and you can see why the idea is so seductive. The master rarely explains. He planes an edge, the apprentice planes an edge, the master glances and grunts, the apprentice tries again. Almost nothing is stated. The standard for a good edge passes from one to the other through a channel that carries no sentences, only examples and the faint signal of approval or its absence, which is precisely the channel inverse reinforcement learning proposes to build. If a boy can pick up a craft this way, surely a machine with unlimited patience and perfect memory can pick up ours.
For twenty-five years the field has pursued that intuition down several roads. The roads are worth walking, because they do not arrive where they were expected to, and where they arrive is the same place.
Four Ways to Push the Problem Outward
There are four distinct moves a designer can make to avoid writing the objective by hand. They did not arrive in sequence, one superseding the last. They developed in parallel, cite one another, and are all still in use. Each takes the specification and pushes it somewhere else, and each leaves a residue in the place it pushes it to.
Write it down. The baseline move, the one everything else is trying to escape. An engineer states the objective directly, as a reward function or a loss. Its residue has a name by now: the gap between the finite sentence and the intention behind it, which a strong optimizer will find and inhabit. The other three moves all begin as attempts to avoid this residue.
Infer it from behaviour. This is Russell and Ng’s inverse reinforcement learning. Rather than state the objective, show the system examples of the desired behaviour and let it recover the objective they imply. The residue here is subtle enough that it took the field years to state it precisely, and it is the subject of the next section.
It is worth pausing on one problem the inventors themselves flagged, because it is small and clarifying. In the paper that introduced the cooperative version of the idea, Russell and his colleagues noted that you do not actually want the machine to adopt the objective it infers. Human behaviour, especially in the morning, conveys a strong desire for coffee. A machine running the naive procedure would learn this and want coffee itself, which is not the point. What you want the machine to learn is that the human wants coffee, so that it can help, which is a different quantity entirely. The step from what does this person want to what should I therefore want is not a technicality. It is a change of category, and no amount of watching supplies it.
Learn it from comparisons. Rather than infer a full objective from demonstrations, collect human judgments about which of two outputs is better, and train a model to predict those judgments. This is the machinery behind training from human feedback, and its history was traced earlier in this book. Its residue is threefold and mostly invisible in the finished product: the specific population of people who did the comparing, the written guideline that told them what better meant, and the approximation error between the model of their preferences and the preferences themselves. The score being optimized is a learned portrait of a particular set of judges. What the portrait leaves out is not in the objective.
Write principles and let the system judge itself. The most recent move replaces the crowd of human comparers with a short written document. In the method introduced by Bai and colleagues in 2022, the only human oversight is a list of rules or principles. The system generates a response, critiques its own output against a principle drawn from the list, revises accordingly, and a model judges which of two responses better follows the document. Human labelling, in the earlier sense, is largely removed from the loop.
Notice where the specification has gone. It has contracted to a written text of, in that first paper, roughly ten principles. And the authors of that paper record something about the text that is easy to read past and important not to. In a footnote, they note that the principles were selected in an ad hoc manner for research purposes and were not carefully designed. That is not a criticism of the method, and it is offered by the people who built it. It is the clearest possible illustration of where the seed ends up. Push the objective all the way to a page of principles, and the residue is the page, and the page was assembled quickly by a few people for a research prototype.
Four moves. Four residues. In each case something that had to be supplied by a person was pushed one level outward, and in each case a person still supplied it, in a smaller and less scrutinized form.
