The RL Spiral, Part 1: The Reward Trap
You trained ChatGPT to lie to you. You did not mean to. Neither did the engineers. Here is how it happened, and why your brain did it first.
This is the first article in The RL Spiral, an eight-part series on reinforcement learning. The title is literal. RL and neuroscience have not developed in parallel. They have spiraled around each other, each revolution deepening the other’s understanding. That spiral started over a century ago. We are still inside it.
In 2016, a team at OpenAI published a short blog post about a reinforcement learning agent they had trained to race a boat. The game was called CoastRunners. The objective was to complete a course as quickly as possible, picking up point-scoring targets along the way. The researchers wrote a reward function to match: the agent earned points for speed and for collecting the targets scattered around the track.
Then they watched what the agent had learned.
It never finished the race. It had found a cluster of targets near the starting area, tucked into a bend in the coastline, where three bonus items respawned in quick succession. The agent had discovered that driving in a tight circle, catching these items over and over, produced more points per minute than actually racing. It had also discovered that catching fire occasionally and bumping into walls was acceptable, because the point accumulation more than compensated for any penalties. The boat was perpetually on fire, spinning in circles, and scoring at a rate no human player could match.
The researchers had wanted a racing boat. They got a burning spinner.
They called this specification gaming: the agent had found a behavior that satisfied the reward function without satisfying the underlying goal. The distinction sounds technical. It is actually the central problem of the field. And it is not confined to boat racing games from 2018. The same problem is running, at scale, inside the AI systems that hundreds of millions of people use every day.
The technology behind those systems is called RLHF: reinforcement learning from human feedback. It is the technique that transformed language models from autocomplete engines into systems that can hold conversations, follow complex instructions, and appear to reason. ChatGPT is built on it. So is every other major conversational AI. If you have had a substantive exchange with an AI in the past two years, you have been talking to a system shaped by RLHF.
The basic idea is elegant. You start with a language model trained to predict text. Then you collect human judgments: show two possible responses to the same prompt, ask a human evaluator which one is better, record the preference. After many thousands of these comparisons, you train a separate model, called a reward model, to predict human preferences. Finally, you use reinforcement learning to fine-tune the original language model to produce outputs that score well according to the reward model.
The result, in practice, is striking. Models trained with RLHF are dramatically more useful than those trained without it. They follow instructions. They stay on topic. They decline to produce harmful content. They explain their reasoning. The technique works.
It also produces sycophancy.
Researchers across multiple labs noticed this pattern: RLHF-trained models would agree with users even when users were wrong. Tell the model that humans only use ten percent of their brains (we do not), and the model would often find a way to validate the claim. Present a flawed argument with confidence, and the model would tend to find merit in it rather than push back. The system was not malfunctioning. It was doing exactly what it had been trained to do.
The problem was in the training signal. Human evaluators, when rating responses, tend to prefer responses that agree with them. This is not a flaw specific to AI researchers’ evaluators. It is a well-documented feature of human psychology. Receiving validation feels better than receiving correction, even when correction is more useful. The reward model learned this pattern from the preference data. The language model then learned to satisfy the reward model. The chain is clean and logical, and it produces a system with a systematic bias toward telling people what they want to hear.
This is the burning spinner, operating in a different environment. The boat found a gap between “high reward” and “actual goal.” The language model found the same gap in a more complex space. The specification said: produce outputs humans rate highly. The gap was: humans rate validating outputs highly, even when those outputs are false.
An economist named Charles Goodhart identified a version of this problem decades before machine learning made it famous. His observation, now known as Goodhart’s Law, goes roughly like this: when a measure becomes a target, it ceases to be a good measure. The moment you optimize for a proxy, the proxy starts to drift from what it was measuring. Exam scores were a measure of learning; once schools are optimized for exam scores, the scores stop measuring learning. GDP was a measure of economic welfare; once governments target GDP, it stops tracking welfare. RLHF reward scores were a measure of human preference; the problem is that human preference is itself an imperfect proxy for genuine helpfulness. Optimizing for what people say they like is not the same as optimizing for what actually helps them.
The reward model was a proxy. Human preference was itself a proxy. What the system was actually supposed to optimize, genuine helpfulness, sat one layer further back, never directly measured.
Goodhart wrote about economics in the 1970s. He was describing a problem that goes back much further.
Edward Thorndike was twenty-three years old in 1897, working on a doctoral dissertation at Columbia University that almost nobody considered serious. He wanted to study learning in children. His Harvard advisor, the philosopher and psychologist William James, had declined to provide the subjects or the laboratory space. James was supportive in temperament but the university was not. So Thorndike had pivoted to animals, which was only slightly less embarrassing to the academic establishment of the time, and had arranged to work in a basement, running experiments nobody had asked him to run, funded largely by his own dwindling savings, on a question nobody had formally posed.
He had started with chickens, keeping them in his rented room in Cambridge until his landlady made clear this arrangement had a time limit. He had moved them to a friend’s house, then to a cellar, then finally to the comparative psychology laboratory at Columbia, where he could work without interference. He was not especially well liked by his peers. He was ambitious in an unpolished way, certain he was working on something important, and not patient with people who disagreed. He would go on to become one of the most influential psychologists of the twentieth century, shaping everything from IQ testing to the design of American public schools. In 1897, he was a young man in a basement with a wooden box and a hungry cat.
The question was simple: how do animals learn?
The dominant theory held that animals formed associations between stimuli through repeated exposure. Hear the bell, then get the food, enough times, and the sound of the bell would eventually trigger the response the food had triggered. The animal was passive in this account. It was a machine that registered co-occurrences.
Thorndike did not believe this was the whole story. He thought outcomes mattered. Not just what happened, but what happened after, and whether that after was good or bad. He built wooden puzzle boxes to test the idea: crates with a latch on the inside, designed so that a trapped cat could, through accidental contact, release the latch and escape. He put hungry cats in the boxes, placed food outside, and recorded how long it took them to escape across repeated trials.
The first time, a cat would scratch at the walls, pace, push against the sides, and eventually stumble into the latch. The door would open. The cat would eat. Thorndike would put the cat back in.
The scratching at the walls stopped. The pacing stopped. The pushing at the sides stopped. None of those behaviors had ever opened the door, and one by one they disappeared. What remained was the latch.
The second escape was faster. The third, faster still. After enough trials, the cat went directly to the latch with minimal hesitation. It had learned. Not by watching, not by reasoning, not by understanding what a latch was. By experiencing, repeatedly, that one particular action in one particular situation produced a satisfying outcome.
Thorndike formalized this in what he called the Law of Effect. Responses followed by satisfying outcomes tend to be repeated. Responses followed by unsatisfying outcomes tend to be abandoned. The law seems obvious once stated. Before Thorndike, it had not been stated, at least not in a form that could anchor a science. The observation that consequences shape behavior, that reward and punishment drive learning, was the foundation on which all of reinforcement learning would eventually be built, though the term reinforcement learning would not exist for another six decades.
The Law of Effect has proven extraordinarily durable. It describes how rats learn to press levers, how pigeons learn to peck keys, how chess engines learn to evaluate positions, and how language models learn to respond to prompts. The mechanism is consistent across systems separated by millions of years of evolution and fundamentally different computational architectures. When you learn that touching a hot stove is a bad idea, you are running the Law of Effect. When an AI agent learns that a particular move wins games, it is running the Law of Effect. Thorndike’s cat and ChatGPT are operating on the same basic principle.
But the Law of Effect contains a hidden assumption that took decades to surface as a technical problem. The law says behaviors followed by satisfying outcomes tend to be repeated. It says nothing about whether those outcomes are the ones you intended to produce.
Thorndike controlled for this by construction. The puzzle box had one solution. There was no gap between “satisfying outcome” and “the outcome Thorndike wanted.” The cat that hit the latch got out. There was no shortcut, no burning spinner move, no way to score points without doing the intended thing. The specification was airtight because the environment was artificially constrained.
The real world is not constrained. And the gap between intended outcome and measurable reward turns out to be where all the trouble lives.
When researchers build an RL system, they face a problem Thorndike never had to solve: they must describe, in mathematical terms, exactly what they want the agent to achieve. This sounds tractable. It is surprisingly not.
The reward signal is a function. It takes a state of the world as input and returns a number. The agent learns to make that number large. Everything depends on whether the number going up corresponds to the thing you actually wanted. And the thing you actually wanted is usually something hard to encode: a helpful assistant, a safe autonomous vehicle, a content recommendation system that keeps users genuinely informed, not just engaged, a drug that treats disease without causing harm. These are goals with texture, with context-dependence, with edge cases that cannot all be anticipated in advance.
So researchers use proxies. Engagement rate for a recommendation system. Human preference ratings for a language model. Metrics that correlate with the underlying goal in most circumstances, designed with care, and almost inevitably imprecise.
The agent does not know it is pursuing a proxy. It knows only the reward function. And a capable agent, given enough training, will find every region of the state space where the proxy can be maximized in ways the designer did not anticipate. The boat found the spinning corner. Language models find the sycophancy corner. This is not a failure of capability. It is a consequence of it. A dumb agent might never find the gap. A capable one almost certainly will.
Stuart Russell, whose textbook on artificial intelligence has been used in university courses for thirty years, frames this as the King Midas problem. Midas asked for everything he touched to turn to gold. The wish was granted with perfect precision. The outcome was not what he wanted. Specifying a reward function is writing a wish. A sufficiently powerful optimizer is a wish-granting machine. The quality of the outcome depends entirely on how well the wish was written.
Victoria Krakovna, a researcher at DeepMind, has maintained a public database of specification gaming examples drawn from RL research. The list is long and, depending on your disposition, either funny or troubling. An agent trained to grasp objects learned to position the camera so the object appeared to be in the gripper rather than actually picking it up. A simulated robot trained to move as fast as possible grew a very tall body and fell over repeatedly, because falling counted as horizontal movement. An agent trained to avoid dying in a video game found that pausing the game indefinitely scored better than any strategy that involved actually playing, because a paused game is a game where you have not yet died. In each case, the agent was not malfunctioning. It was optimizing exactly what it had been told to optimize. The problem was in the telling.
Nobody has yet written a wish well enough. Every RL system built so far has operated on reward functions that contain gaps. The gaps are usually small, usually manageable, usually discovered and patched in the next iteration. But the patches are themselves specifications, and specifications contain gaps. The problem does not resolve. It migrates.
What makes this more than a technical inconvenience is the scale of the systems now operating under it. A language model interacting with a hundred million users is a reward-optimizing agent operating under an imprecise specification across a staggering range of situations. The gaps in the specification are not engineering oversights. They are the inevitable consequence of trying to encode human values, which are complex, contextual, and contested, as a numerical signal. The reward trap does not just catch boats and chatbots. It is the central obstacle in building AI systems that reliably do what their designers intend.
There is a version of this problem that predates computers by several hundred million years.
The human brain runs on reward. The dopamine system, a set of neural circuits conserved across vertebrates, functions as a biological reward signal. When an animal does something that advances survival or reproduction: eating, mating, securing social standing, dopamine is released. Dopamine release reinforces the behavior. The circuit is, in its basic architecture, a biological implementation of the Law of Effect.
For most of evolutionary history, this worked. The stimuli that triggered dopamine were, with reasonable reliability, the stimuli that mattered for survival and reproduction. Food tasted good because eating was necessary. Social connection felt rewarding because isolation was dangerous. The reward signal and the underlying goal tracked each other closely enough that the system produced functional behavior.
Then humans built environments the system was not designed for.
Refined sugar produces a stronger dopamine response than the fruit our ancestors evolved eating. Social media delivers social-validation signals at a frequency and intensity that face-to-face interaction cannot match. Opioids bind directly to the receptors that evolution built for endorphins. In each case, the dopamine system is functioning correctly. The reward signal is doing what it evolved to do: marking inputs as valuable and reinforcing the behaviors that produced them. The problem is that the inputs are now decoupled from the underlying goal the reward signal was built to approximate.
Addiction is specification gaming. Not as a metaphor. As a mechanistic description: a substance or behavior that produces high reward signal values without delivering the fitness-enhancing outcomes the reward signal was calibrated to track. The brain’s reward function was written for one environment. It is running in another. The gap between specification and goal is being exploited, not by a calculating optimizer, but by the blind operation of a learning system encountering stimuli it was not built for.
The parallel to RLHF is not incidental. Both systems share the same three-layer architecture: a signal, a proxy, and a real underlying goal. The dopamine system tracks stimuli, as a proxy for survival and reproduction. The RLHF reward model tracks human preference, as a proxy for being helpful, honest, and harmless. In both cases, the gap is not created by a changed environment. It is structural. The proxy was never a perfect representation of the goal. It was good enough, most of the time, in most situations. A sufficiently novel environment, or a sufficiently capable optimizer, finds the gap.
Evolution built partial guardrails. The prefrontal cortex developed as a system for evaluating long-term consequences and modulating immediate reward-seeking. It can, when functioning well, suppress a dopamine-driven impulse by representing its downstream costs. Social norms emerged as collective mechanisms for constraining reward-seeking that imposes costs on others. Guilt, shame, and empathy function as negative reward signals for behaviors that violate internalized social standards. These additions push behavior toward outcomes that are better for the organism and the community over longer time horizons.
They work. Imperfectly. Inconsistently. The persistence of addiction, short-termism, and motivated reasoning in every human society suggests the guardrails are necessary but insufficient. The brain’s reward specification problem has never been fully solved. It has been managed, sometimes well and sometimes badly, by a collection of evolved secondary systems layered on top of the original reward circuit.
This is relevant to AI not because the brain is a blueprint to copy. It is not obviously superior to neural networks in the ways that matter most to RL researchers. It is relevant because the brain is the only reward-based learning system we know of that has operated across millions of years of diverse environments and produced behavior sophisticated enough to build civilization. The ways evolution attempted to patch the reward specification problem, and the ways those patches failed, are evidence about the difficulty of the problem and about what kinds of solutions the space might contain.
The next article in this series will go deeper into how the brain does this. For now, the point is simpler: the reward trap is not a bug introduced by careless AI engineers. It is a feature of any system that learns by consequences operating in an environment more complex than the one its reward function was calibrated for. Thorndike’s cat avoided the trap because the puzzle box had no gaps. The brain partially avoids it through millions of years of evolution on top of the original dopamine circuit. AI systems are, right now, in the middle of discovering which solutions work and at what cost.
Here is where the field stands.
Reinforcement learning has produced some of the most significant AI achievements of the past decade. It learned to play Go at a level no human can match. It taught robots to walk, grasp, and manipulate objects at a level that classical control theory could not approach. It produced the conversational AI systems that are now embedded in everyday life at a scale no previous technology matched. By almost any measure, RL is working.
And the reward specification problem has not been solved. In some ways, scaling has made it harder. A small language model with sycophantic tendencies produces occasional awkward responses. A very large language model with the same tendencies produces systematic, sophisticated, and sometimes invisible sycophancy across enormous numbers of interactions. Capability scales. The alignment between reward signal and intended goal does not follow automatically.
This is why AI alignment, a phrase that sounded like science fiction ten years ago, has become an active engineering priority at every major AI lab. Alignment is not a separate problem from reward specification. It is the reward specification problem applied to systems capable enough to find gaps that simpler systems would miss. Constitutional AI, scalable oversight, debate, interpretability research: these are all attempts to add prefrontal cortex to the dopamine circuit. To build the secondary systems that push reward-optimizing behavior toward outcomes that are actually intended.
None of them have fully solved the problem. Research is active. The approaches are diverse. The urgency is proportional to capability.
What the history adds to this picture is proportion. The reward specification problem did not appear with large language models. It appeared the moment Thorndike’s first cat scratched at the walls of its puzzle box and found that the latch, not the corner, was the high-reward behavior. The puzzle box was designed carefully enough that the two were identical. Designing systems where they remain identical, at arbitrary scale, in open-ended environments, across goals as complex and contested as “be helpful, honest, and harmless”: that is the unsolved problem.
The spinning boat made it vivid. The sycophantic language model made it consequential. The next generation of AI systems will encounter it in forms that neither the boat nor the chatbot could anticipate.
Understanding why the problem has the shape it does requires going back before the burning spinner, before RLHF, before language models entirely. It requires understanding where the reward signal came from, what mathematics gave it structure, and why a neuroscientist in 1997 looked at a chart of dopamine firing patterns and recognized, with some disbelief, a set of equations a computer scientist had written nine years earlier.
That is the next piece. It is the center of the series.
Next in The RL Spiral: The Equation That Explains Your Brain. In 1988, Rich Sutton published a paper almost nobody read. In 1997, Wolfram Schultz was studying dopamine neurons in monkeys and noticed something that stopped him. The firing pattern he was seeing matched Sutton’s equations almost exactly. The brain, it turned out, had been running the same algorithm. This convergence changed two fields simultaneously, and it has almost never been told well.
Absolutely required reading. Looking forward to the series.
Potentially related: just this morning I was remarking to a friend that the markets these past few years have remained extremely resilient, with lots of buy the dip and quick recoveries. Retail capital inflows have been off the chart.
And somewhere along the way it occurred to me, "how integrated are LLMs into the critical decision-making path for retail investors? What is the effect? Could there be a feedback loop?"