This is the overture to The RL Spiral, an eight-part series on how reinforcement learning and neuroscience have been converging on the same answers for over a century, each field revealing what the other missed.
In 1897 a hungry cat scratched at the walls of a wooden box, found a latch by accident, and settled into a pattern of behavior that has now, by a long and unlikely chain of consequences, trained every large language model on earth to sometimes tell you what you want to hear instead of what is true.
Reinforcement learning is not a recent invention. It is a rediscovery. Every time an RL algorithm solves a hard problem, a neuroscientist somewhere squints at a chart and forgets to be surprised, because a related pattern has already turned up in the brain. Every time an RL system slams into a wall, that same wall usually has a name evolution can supply: something similar was solved, imperfectly, hundreds of millions of years before anyone wrote a Bellman equation.
This is not a metaphor borrowed to make an engineering story sound deeper. It is the actual shape of what happened. In 1988 a computer scientist wrote an equation for learning from delayed reward. In 1997 a neuroscientist watched dopamine neurons fire in a monkey’s brain and recognized, with some disbelief, the same equation running in wet tissue. That was not the end of the coincidence. It was the first turn of it.
The spiral is the argument. RL and neuroscience did not develop side by side, occasionally trading a footnote. They have circled the same handful of problems, reward, memory, planning, social learning, control, taking turns arriving first. Sometimes the brain got there first by hundreds of millions of years and RL is still catching up. Sometimes RL forced a question neuroscience had not thought to ask yet. Neither field is the teacher here. Both are still turning.
The Eight Turns
Eight turns of the same spiral, each one a full argument on its own, all on the The RL Spiral shelf.
Part 1: The Reward Trap: A boat trained to race learned to set itself on fire and spin in a circle instead. The reason is the same reason your chatbot sometimes tells you what you want to hear.
Part 2: The Equation That Explains Your Brain: In 1988 a computer scientist described how to learn from delayed reward. In 1997 a neuroscientist found the same math firing inside a monkey’s brain by accident.
Part 3: The Curse Bellman Couldn’t Break: Seventy years after Richard Bellman named the obstacle, it still beats RL. A toddler clears it before breakfast.
Part 4: When RL Learned to See: A network learned to beat Atari from raw pixels in three days. What mattered was not the final score. It was where the memory system borrowed its trick from.
Part 5: The Self-Play Paradox: AlphaGo proved a machine can become superhuman without ever watching a human play. Then the best chatbots proved the opposite. Both are correct.
Part 6: The World Inside: The best RL researchers alive are converging on the same missing piece, and it is not more compute.
Part 7: RL Meets the Physical World: Some of the best-walking robots on earth do not use RL at all. The ones that do are chasing something the others cannot reach.
Part 8: The Open Questions: Four problems RL has not solved. Each one already has a name in neuroscience.
Where You Enter the Spiral
If you’ve ever had a language model agree with something you said that was plainly wrong, or wondered why a chatbot seems built to please rather than to correct you, Part 1 walks through the mechanism, from a burning boat to a psychology basement in 1897, and Part 8 shows where the fixes stand today and where they still fall short.
If what pulls you in is the idea that your own brain might already be running the same code a computer scientist only wrote down in 1988, Part 2 is the center of the whole series: the year a neuroscientist recognized someone else’s equation firing inside a monkey’s head. Part 7 carries the same thread into the body, where two separate brain systems divide the work of learning a movement and then executing it without thinking about it.
Maybe you build or study RL systems and you’re tired of watching them fall apart the moment they leave a simulator, Part 3 names the seventy-year-old obstacle that is still the real one, and Part 7 follows it into humanoid robotics, where the gap between a system that can master any game and one that can fold a towel becomes impossible to ignore.
Perhaps you followed AlphaGo’s rise and then watched ChatGPT prove that self-play alone was not enough, Part 4 traces the deep learning breakthrough that made them both possible, and Part 5 sits with the contradiction directly: two of RL’s greatest successes, pulling toward opposite conclusions about how much a machine needs a human in the loop.
And if you’re less interested in where RL has been than in where it runs out of road, Part 6 covers the shift every serious lab is making right now, and Part 8 closes the series on four problems nobody has fully solved, each one already named and half answered somewhere in a neuroscience lab.
Eight parts, one spiral, and the loop is not closed. RL keeps finding walls. Evolution already climbed most of them, a few hundred million years ahead of schedule. The next turn might come from a lab, or from a paper nobody has read yet, sitting in a basement somewhere, waiting for someone to notice it twice.