Tell the Machine What, Not How
A cat in a wooden box. A century of punched cards. A machine that taught itself checkers. Somewhere in there, we stopped telling machines how.
From Whatever You Ask For, a series on the one job machines cannot do for us.
A Cat and a Fish
In a room at Columbia University in the late 1890s, a graduate student named Edward Thorndike built a series of small wooden crates and put hungry cats inside them.
The crates were crude. Slats, a door held shut by a simple catch, and somewhere within reach a loop of string or a lever or a treadle that would release it. Outside the door, in plain view and plain smell, Thorndike placed a scrap of fish.
The first cat did what any animal does when it finds itself in a box. It clawed at the slats. It pushed its paws through the gaps. It bit things. It thrashed around with no plan whatsoever, and after roughly a hundred and sixty seconds of this, it happened to strike the catch. The door swung open. The cat got the fish.
Then Thorndike picked the cat up and put it back in the box.
Thorndike did not show the cat the latch. He did not guide its paw. He did not, in any sense available to a cat, explain the mechanism of the door. He had no way to communicate a procedure and no intention of trying. All he did was run the same situation again, and again, and again, recording with a stopwatch how long each escape took.
By the twenty-fourth trial, the same cat in the same box was out in about six seconds.
Something had been transmitted from the experimenter to the animal. It clearly was not a method, because no method was ever sent. What Thorndike had supplied was narrower and stranger than a method. He had supplied a definition of success. Fish on the far side of the door, and nothing else in the universe of that box mattered. Everything the cat eventually knew about latches, it worked out for itself, from that one piece of information, delivered over and over.
Thorndike wrote this up in 1898 as his doctoral dissertation, under the title Animal Intelligence: An Experimental Study of the Associative Processes in Animals. The principle he extracted from those escape times became known as the law of effect, and by the time he set it out formally in his 1911 book, it had a shape that would outlive him by a century: responses followed by satisfaction become more likely to recur in that situation, and responses followed by discomfort become less likely.
Read as psychology, this is a claim about animals. Read as engineering, it is something else. It is a recipe for producing intelligent behavior in a system without knowing how that behavior works. And it points at a fork that no one at the time had any reason to notice, because only one of the two roads existed yet.
Here is the fork. If you want a system to do something, there are exactly two things you can give it.
You can give it the method. Do this, then this, then if the light is red do that. This is the road of instructions, and it requires that you already know how to do the thing.
Or you can give it the criterion. Here is what counts as success. This is the road Thorndike stumbled onto with a wooden box and a piece of fish, and it requires almost the opposite: you do not need to know how to do the thing at all. You only need to be able to recognize it when it is done.
For the next fifty years there were no machines to take the second road with. When machines finally existed, nearly all of the effort went down the first road, and it went down it with such spectacular success that the second road stayed a footnote for another half century.
The fork is not only history. It describes the arrangement you personally are living inside right now, and describes it better than any of the words currently used for it.
The Century of Instructions
The road of instructions is one of the great achievements of human civilization.
Begin in Lyon in the first years of the nineteenth century, with Joseph Marie Jacquard and a loom. Weaving a figured pattern into silk had until then been a matter of a skilled weaver and an assistant, often a child, physically lifting warp threads in the correct combination, row after row, from memory or from a written pattern, for weeks. Jacquard’s loom read the pattern off a chain of stiff cards, each card punched with holes in the positions where threads should rise. Rods pressed against the card. Where there was a hole, a rod passed through and lifted its thread. Where there was no hole, the rod was blocked.
Look at what has happened here. A pattern in a weaver’s head, which is to say a procedure, has been turned into an object. A physical, countable, stackable object that can be carried across a room, copied, sold, stolen, and executed by a machine that understands nothing about silk. The cards for an elaborate design ran into the thousands, and the celebrated woven portrait of Jacquard himself required something on the order of twenty thousand of them, a chain of punched cardboard feeding through the machine and producing, at the far end, a face.
Charles Babbage owned one of those woven portraits. Ada Lovelace, writing about Babbage’s proposed Analytical Engine in 1843, made the comparison explicit and famous: the engine weaves algebraic patterns the way the Jacquard loom weaves flowers and leaves. She was not being poetic about it. She was pointing out that the same trick works on numbers, and in the notes accompanying that observation she wrote out what is generally regarded as the first published algorithm intended for a machine.
A century later the trick had a body. The stored program computer of the late 1940s took the final step of putting the instructions in the same memory as the data, which meant a program could be written, edited, copied, and even modified by another program. What followed was the entire discipline of software: an industrial civilization built on the premise that human procedural knowledge can be extracted from human heads, written down with total precision, and executed at inhuman speed by a machine with no understanding of what it is doing.
By the 1960s this premise had produced payroll systems, air traffic control, guidance for spacecraft. It is difficult to overstate how well it worked, and worth stating plainly that most of the digital world you touch daily is still built this way and works beautifully.
But there was a ceiling, and the ceiling was always there from the very first card.
The road of instructions requires that somebody, somewhere, can say how. Not vaguely. Exactly. Every branch, every condition, every case. If a procedure cannot be articulated to that standard, it cannot be punched into a card, and the machine cannot execute it.
For a while this seemed like a manageable constraint, a matter of effort. It is not. In 1966 the chemist and philosopher Michael Polanyi put his finger on the reason in a sentence that has been quoted ever since: we can know more than we can tell.
Consider what you know how to do. You can recognize your mother’s face in a crowd in a fraction of a second. You cannot write down the procedure. You can ride a bicycle, an activity involving continuous corrective steering into the direction of fall, and if you try to explain it to a child you will find yourself saying words that are true and useless. You can read a room. You can tell when a sentence sounds wrong. A physician of thirty years’ experience can look at a patient and feel that something is off before any test comes back, and will often be right, and will often be unable to say why.
None of this is mystical. It is simply that the overwhelming majority of what a competent human being knows was never in the form of stated rules to begin with. It was learned the way Thorndike’s cat learned the latch, by doing and by consequences, and it lives in a form that does not decompose into steps.
So the instruction road, for all its power, could only ever reach the things we can say. And the things we can say turn out to be a strange, thin slice of what we know. We can say how to calculate a payroll. We cannot say how to see.
This was the wall that the first great era of artificial intelligence spent decades running into. The effort to write down expertise as explicit rules produced systems that were impressive in narrow domains and brittle everywhere else, and the brittleness was not a bug to be fixed by more rules. It was the ceiling, showing through.
Getting past it required going back to the fork and taking the other road. And the other road had been sitting there, largely unnoticed by anyone building computers, in the psychology department.
The Other Bloodline
Thorndike’s line of descent did not run through engineering. It ran through animal laboratories, and for the first half of the twentieth century it had nothing to do with machines at all.
B. F. Skinner extended the work in the 1930s with a rigor that made it a technology. If behavior is shaped by consequences, then consequences can be scheduled, and if they can be scheduled they can be engineered. Skinner could take a pigeon and, by delivering food at precisely chosen moments, build behavior that no pigeon had ever performed. Turning in a circle. Pecking a specific key when a specific light was on. Complex sequences assembled piece by piece, each step reinforced until it was reliable, then withheld until the next step appeared.
At no point did Skinner tell a pigeon anything. He had, as Thorndike had, exactly one channel of communication with the animal, and that channel carried one kind of message: that was good, do more of that.
During the Second World War, Skinner took this to a conclusion that is either absurd or visionary depending on how you look at it, and is probably both. Guided missiles did not yet exist in usable form. Skinner proposed to solve the guidance problem with pigeons. Birds would be trained to peck at the image of a target on a screen in the nose of a glide bomb, and their pecking, mechanically coupled to the control surfaces, would steer the weapon in. The project was real, it was funded, and the pigeons worked. They pecked accurately through noise, through motion, through everything the researchers threw at them. The military never deployed it, which is unsurprising, and the demonstration stands anyway: an animal trained purely by consequence had become a functioning component in a control system.
The first person to see clearly that this could be pointed at computers was Alan Turing.
His 1950 paper in Mind is famous for the imitation game, and the final section, where the real argument lives, is read far less often. Turing has spent the paper arguing that a machine might think. He arrives at the practical question of how one would ever build such a thing, and he rejects the obvious approach. Do not try to program an adult mind, he says. Programming an adult mind means writing down everything an adult knows, and we have already seen where that road ends. Instead, build something with the structure of a child’s mind and then educate it.
And when he asks what education consists of, he reaches, without apparent hesitation, for Thorndike’s channel. Turing proposes that the machine be built so that events preceding a punishment signal become less likely to repeat, while a reward signal raises the probability of repeating whatever led up to it. He notes that this framing does not require the machine to feel anything.
Then comes the line that gets left out of the summaries. Turing writes that he has done some experiments with one such child machine and succeeded in teaching it a few things, but that his teaching method was <q>too unorthodox for the experiment to be considered really successful</q>.
He tried it. In 1950, with essentially no hardware and no theory, the man who defined computation sat down and attempted to raise a machine by reward and punishment, and reported that it had not gone well. Almost everything that would matter seventy years later is already present in that paragraph, including the failure mode.
Turing also spotted the limit immediately, and it is the same limit that governs the field today. If reward and punishment are your only channel to the learner, then the total information you can transmit is bounded by the number of rewards and punishments you deliver. A single bit at a time is a narrow pipe. You will need other channels, he says, and suggests language.
Nine years later, at IBM in Poughkeepsie, Arthur Samuel built the thing.
Samuel chose checkers, a game with simple rules and a search space large enough that no one can hold it in a head. His program, described in his 1959 paper in the IBM Journal of Research and Development, searched ahead through possible moves and evaluated the resulting positions using a scoring function assembled from board features. That much was ordinary engineering of the instruction kind. The scoring function had adjustable weights, and Samuel could have set them by hand, as everyone else would have.
What he did instead was let the program play against itself, and adjust its own weights based on how the games came out.
Sit with the strangeness of that. Samuel wrote down the rules of checkers, which he knew, and a definition of winning, which he knew, and then he stopped writing. He did not supply the judgment of which positions are strong. He supplied the criterion and let the machine work backward to the judgment, overnight, on an IBM machine, playing itself in an empty building. The program’s positional sense came out better than the one Samuel could have tuned by hand. He had built a system that exceeded his own ability to specify it.
That paper is also where the term machine learning entered wide circulation.
The program’s later history is a caution rather than a triumph. In 1962 it won a single game against Robert Nealey, a strong Connecticut player, and the press converted a single game against a state-level opponent into a story about checkers being solved and computers surpassing all human players. Neither was remotely true, and the myth did real damage, steering serious research away from the game for a quarter century. The lesson to take is not about checkers. It is that the moment a machine learns something we did not teach it, our instinct is to wildly overread what happened, and that instinct has not improved since 1962.
Look back along this line. Thorndike with his fish, Skinner with his pigeons, Turing with his unorthodox and unsuccessful lessons, Samuel with his empty building. Every one of them is performing the same refusal. Each of them declines to supply the method. Each of them supplies only a signal for what counts as good, and lets the system find its own way there.
For sixty years that refusal was a curiosity. Then it became the main line of artificial intelligence, and the question of what exactly gets put into that signal became the most consequential question in the field.
One Axiom
Somewhere in the middle of the twentieth century, the second road acquired a formal skeleton, and the skeleton is simple enough to state in a paragraph.
There is an agent. There is an environment the agent sits inside. At each moment the agent observes something about the state of the environment, takes an action, and receives from the environment a number. That number is called the reward. The agent’s entire purpose is to act so as to maximize the total reward it accumulates over time. It is not told which actions are good. It has to work that out from the numbers.
Three pieces. Agent, environment, reward. That is the whole apparatus.
The audacious part is not the framework. It is the claim made on the framework’s behalf, which Richard Sutton has stated and which is known in the field as the reward hypothesis: that all of what we mean by goals and purposes can be well thought of as maximization of the expected value of the cumulative sum of a received scalar signal.
All of what we mean by goals and purposes. Not some. Winning a game, yes, obviously. But also folding a protein, driving to an address, keeping a data center cool, writing a sentence a reader will find helpful, and, if the hypothesis is taken at its word, raising a child well and living a decent life. The claim is that every one of these, however rich and textured it feels from inside, can be captured without essential loss as a single running number to be made as large as possible.
Sutton has compared the status of this claim to the expected utility hypothesis in economics, and the comparison is apt in both directions. It has organized an enormous amount of productive work. It is also the sort of claim that provokes immediate objection from anyone who hears it for the first time, and the objections are not stupid.
For now, accept it. Take it as an axiom and see what follows, which is what the field did. The reason to accept it provisionally is not that it is obviously true. The reason is that it has been by far the most productive assumption anyone has made about machine intelligence, and a claim that productive has earned the right to be taken seriously all the way to its conclusions, including the uncomfortable ones.
One conclusion is available immediately.
If the reward hypothesis holds, then building a capable system decomposes into two jobs of wildly unequal glamour. Job one is figuring out how to maximize a given reward. Job two is deciding what the reward is.
Job one is what the whole field works on. It is where the algorithms are, the compute, the papers, the money, the talent. It is very hard and the progress on it has been extraordinary.
Job two is a single line of code, or a sentence in a spec, or a choice so obvious that nobody records having made it. It takes an afternoon. And it is the only input the human side of the arrangement actually provides.
This is not an abstraction. You can watch it operate at street level.
Consider a food delivery platform. The rider is not given a method for delivering food, because no one can write one. What the rider is given is a criterion, and the criterion is largely time. Meituan has explained publicly that the promised arrival time for an order is computed by taking the longest of several algorithmic estimates and adding a buffer, and the platform’s dispatch and pay have historically leaned on whether that time is met. Everything the rider does, every route, every decision about which order to take and which stairwell to run up, is worked out by the rider from that one signal.
And so riders learned the latch. They learned it exactly as well as Thorndike’s cat, and the thing they learned to do was ride the wrong way up one-way streets and cross against red lights, because the clock was in the objective and their own safety was not. This was documented in detail in a 2020 investigation by the Chinese magazine Renwu under the title “Delivery riders, trapped in the system,” and reported subsequently by Caixin and others. The platforms have since moved on the mechanism, with Meituan announcing in 2025 that it would phase out late-delivery deductions for crowdsourced riders in favor of a points-based scheme.
Nothing in that story is a malfunction. The optimizer worked. It maximized what it was given, at the expense of everything it was not given, and there is no version of a sufficiently strong optimizer that behaves otherwise. The mismatch was not in the optimization. It was in the specification, which took an afternoon, and which no one thought of as the hard part.
Which is the point. When the machine’s job is to maximize, the human’s job is to decide what. And a system that optimizes hard will find every gap between what you wrote and what you meant.
The Handover No One Signed
Step back and look at the two roads together, and at the direction of traffic.
From Jacquard’s cards to the software industry, the human supplied the method. The machine supplied speed and tirelessness and precision. The division was clear, and in that division the human contribution was enormous. Writing down how to do things, exactly, for machines that understand nothing, is most of what a century of engineers did with their working lives.
From Thorndike’s box to the systems now being built, the human supplies the criterion. The machine works out the method itself, and increasingly works it out better than the human could have. In that division the human contribution has become very small and very concentrated. It has narrowed to a single question, asked once, usually quickly: what counts as success?
This handover has already happened across most of the territory where it matters. It happened without an announcement, because it was not an invention with a date. It was a change in the division of labor, made piecemeal, by thousands of people who were each just choosing a loss function or a metric or a target and getting on with the interesting part. Nobody signed anything.
And it left the human side of the arrangement holding a role that has no name.
You can see the role clearly in Thorndike, who never taught a cat anything and simply decided that fish on the other side of the door was what counted. You can see it in Samuel, who did not know good checkers judgment and did not need to, because he knew what winning was. You can see it in whoever, at some point, wrote down that a delivery is on time or it is not.
Every optimizing system in the world has one of these people behind it. Someone chose the number. That choice was invisible because it looked trivial next to the machinery it set in motion, and because our vocabulary has no word for the person who makes it. Call them the reward giver.
It is not a job title. It is a description of what the human half of every one of these arrangements now does, and there are far more people doing it than realize they are. Every manager who sets a target for a team is doing it. Every teacher who decides what the grade rewards is doing it. Every parent who decides which behaviors in a household get warmth and which get friction is doing it, and doing it to a learning system considerably more capable than any of the ones described above.
You have been doing it too, in the arrangement that matters most and gets examined least, which is the one where you are both the reward giver and the optimizer. You are extremely good at maximizing. Whatever you have actually been treating as the score, you have been climbing it for years with real ingenuity, and you have almost certainly found some latches along the way that you are not proud of.
Which raises the question the rest of this book is for.
When you had to supply the method, a bad choice of goal was buffered by all the work of execution. There was time to notice. Now that the method comes for free, the specification is the whole of the human input, and everything downstream of it is fast, tireless, and indifferent.
When did you last check what you wrote down?
This is Part 1 of Whatever You Ask For, a series on the last thing machines will ever need from us, and how badly we do it.