The Journey of RL, Part 12: Learning What to Optimize
Six open problems define the next decade. None of them is “better policy optimization.”
In 1898, a cat sat in a wooden box on a bench at Columbia University, and a graduate student named Edward Thorndike held a stopwatch. The cat wanted the food outside the box. To reach it, the cat had to pull a loop of string that opened the door, and at first it had no idea this was so. It scratched, it bit, it pushed at the slats, and eventually, by accident, it caught the string and the door fell open. Thorndike wrote down the time. He put the cat back in. Over two dozen trials, the chaos narrowed toward a single efficient motion, and a cat that had once taken three minutes was out in six seconds. Thorndike had his graph, bending downward over trials, and he had the beginning of a science: behavior shaped by its consequences, the response that led to satisfaction growing stronger each time.
This series began there because everything that followed inherited the cat’s situation. The cat learned what to do. It did not have to learn what escaping was worth, or whether the food was the right thing to want, or how to weigh the food against the effort or the fear. The world supplied all of that directly. Freedom and food were simply good, unarguably, and the only question was which action produced them. For most of the century that followed, this was the shape of the problem reinforcement learning inherited: the reward was given, a fact about the world as plain as the food outside the box, and intelligence was the search for the actions that earned it.
The eleven parts before this one have been the story of what happened when that plainness dissolved. As reinforcement learning left the world of games and mazes, where the reward really was given, and entered the world of things people actually care about, the reward stopped being a fact and became a problem. It had to be shaped, then constructed, then learned from human judgment, and every one of those moves opened a gap that optimization could exploit, until the field found itself in the crisis this series has spent its second half tracing. The verifiable turn of the previous part was the closest thing to an escape, and it was real, but it was bounded. And so we arrive at the last part with a question that Thorndike’s cat never had to face, and that the field can no longer avoid. Not how to optimize. What to optimize. This part is about the six versions of that question that will define the next decade, and about why none of them is a question the previous eleven parts already answered.
