This is the overture to The Journey of RL, a twelve-part journey across reinforcement learning told through one core question: how did machines learn what to optimize?
In 1898, a psychologist named Edward Thorndike built a wooden box with a latch, put a hungry cat inside it, and left a piece of fish just out of reach on the other side of the bars. The cat clawed at the box with no particular plan until, by accident, its paw caught the latch and the door swung open. Thorndike ran the experiment again, and again. Each time, the cat took a little less time to find the latch. He called this the Law of Effect: behavior followed by satisfaction becomes more likely, behavior followed by discomfort becomes less likely. He was describing a cat. He had no idea he was writing the founding assumption of a field that would not exist for another sixty years.
Reward, in reinforcement learning, is usually presented as a mathematical primitive: a number the agent is built to maximize. It is not one. It is an inheritance from behaviorist psychology, and it carried that lineage’s assumptions into computer science whether the field chose them or not. Christopher Watkins formalized Q-learning in 1989 without needing a model of the environment. Sutton and Barto’s textbook, first published in 1998, stated the central proposition as though it were obvious: everything an agent optimizes for can be collapsed into a single scalar signal. Nobody was lying. Nobody had checked.
The interesting question was never whether reinforcement learning works. It obviously does. The interesting question is what the field was forced to admit about reward each time that founding assumption stopped holding, from Watkins’s convergence proof to the verified rewards now training the reasoning models answering your questions.
Twelve parts split cleanly down the middle. The first six treat reward as an axiom, the given, the fixed point everything else optimizes against. The last six treat reward as the problem itself, something that has to be rescued, replaced, or earned back from scratch. Part 6 sits at the hinge. This is not a tribute. The dead ends get as much room as the breakthroughs, because in this history, reward kept dying and coming back wearing a different name.
Axiom to Problem
Twelve parts, six and six, all filed under The Journey of RL:
Part 1: Before the Equation: the borrowed 1898 assumption that reinforcement learning never got around to questioning.
Part 2: The Value Hypothesis: how collapsing everything an agent learns into one number became the field’s biggest win and its quietest trap, in the same move.
Part 3: Policy or Value: two routes to the same destination, and why the less elegant one is the one that survived contact with neural networks.
Part 4: The Exploration Question: what an agent has to figure out before it can optimize anything at all.
Part 5: The Internal Model: what changes once an agent stops needing the outside world to tell it whether it is doing well.
Part 6: The Brain in the Loop: forty years of reinforcement learning and neuroscience reading each other’s mail, told without assuming you already know the correspondence.
Part 7: The First Cracks: three separate attempts to rescue reward, and three separate confessions about what was wrong with it.
Part 8: The Human Turn: the moment reward stopped being given and started being learned instead.
Part 9: The Goodhart Collapse: why a learnable proxy for a value was never going to survive contact with an optimizer built to exploit it.
Part 10: When Scale Replaced Signal: what the field did when fixing reward looked harder than fixing the data and the compute instead.
Part 11: The Verifiable Turn: the escape from the Goodhart collapse, and the old trick that turned out to be hiding inside it.
Part 12: Learning What to Optimize: six open problems for the next decade, and why not one of them is a better optimizer.
If This Is You
If you use RLHF, PPO, GRPO, or TD-learning without a throughline connecting any of them, start with Part 1: Before the Equation and Part 2: The Value Hypothesis. The second one is where the field’s founding compression and its founding trap get made in the same equation.
Maybe you remember “RL is dead, just use supervised learning” being a sentence people said out loud, not a strawman. Part 7: The First Cracks and Part 8: The Human Turn are the two parts where that sentence gets answered, first by admission, then by RLHF.
If your interest runs from brains toward machines rather than the other way around, Part 6: The Brain in the Loop retells forty years of RL-neuroscience conversation from the ground up, with no prior reading of The RL Spiral required.
Perhaps reward hacking, sycophancy, and alignment tax are words you use constantly without knowing exactly where the trouble started. Part 9: The Goodhart Collapse is where it started, stated as precisely as the field has ever managed to state it.
And if you want the view from the front edge of the field rather than its history, Part 11: The Verifiable Turn and Part 12: Learning What to Optimize cover RLVR and the six open problems defining the next decade, and not one of those problems is a better optimizer. Every route through this series ends at the same place: reward was never as given as it looked.
The cat found the latch by accident. Everything since has been the argument over what should count as finding it on purpose.