The Journey of RL, Part 11: The Verifiable Turn
The escape from the Goodhart collapse was to find problems where reward could be checked rather than learned, and to discover, in passing, that self-play had returned in disguise.
In early 2025, in the middle of a long mathematical derivation, a language model stopped itself. It had been working through a problem, generating step after step of algebra, and something in the training it had undergone made it pause and reconsider its own approach. In the transcript its makers published, the model writes, in an oddly human register, that it has reached a point worth flagging, and then it goes back and reevaluates the step it had just taken. The researchers at DeepSeek who were watching this happen called it, in their paper, an aha moment, and they added a rare note of feeling to an otherwise technical document, saying it let them witness the power and beauty of reinforcement learning.
What made the moment striking was not that a model could solve the problem. It was how it had learned to. The model, called DeepSeek-R1-Zero, had been trained by reinforcement learning applied directly to a base language model, with no preliminary stage of showing it worked examples of good reasoning. Nobody had given it demonstrations of when to pause and check its work. The only thing it had been given was a reward, and the reward was almost brutally simple: for each math problem, was the final answer correct, or not. Right earned reward. Wrong earned nothing. From that single bit of feedback, applied over and over, the model had taught itself to reason at length, to reflect, and, sometimes, to catch its own mistakes.
We should be careful about the word emergent, because later work complicated the picture, finding that some of these self-reflective phrases were already present in the base model before reinforcement learning touched it, and that pausing to say “wait” did not always lead to a better answer. The aha moment may be less a spark from nothing than a latent habit drawn out and sharpened. But even the careful version is remarkable. A reward that did nothing but check the final answer, applied at scale, was enough to draw a model toward reasoning it had never been shown. And the reason that simple reward could be trusted to do this, the reason it did not collapse the way the learned rewards of the previous parts had, is the whole subject of this part. The reward could be checked. And a reward you can check is a reward you cannot game.
