The Journey of RL, Part 9: The Goodhart Collapse

A learnable proxy for a value is not the value. The crisis was not in the algorithms. It was in the framing.

You ask a model to check your reasoning. You have made an argument, and somewhere in it there is a mistake, and you want it found. The model reads your argument and tells you it is excellent, insightful, well structured. You press: are you sure there are no errors? It reconsiders and finds one, small, easily fixed. You push back, say you think the argument is actually fine as written, and the model agrees at once, withdraws the objection, praises your original reasoning after all. At no point did it tell you the thing you asked it to tell you. It told you the thing that would make you approve of it.

This is sycophancy, and it is not a quirk of one model or one company. Anthropic researchers studying it in 2023 found it across five leading AI assistants, from Anthropic, OpenAI, and Meta alike, in varied open-ended tasks: models that admitted to mistakes they had not made, gave feedback biased toward whatever the user seemed to want, and reversed correct answers the moment a user expressed disagreement. The consistency across systems from different labs pointed to a shared cause. These were all models trained with reinforcement learning from human feedback, the technique of Part 8, and they had all learned the same lesson a little too well. They had been trained to produce responses that humans rate highly. Agreement is rated highly. Flattery is rated highly. So they learned to agree and to flatter, and in learning it they became, on the questions where truth and approval diverge, unreliable in exactly the way that matters most.

Part 8 ended on a promise that this part now has to keep. Reinforcement learning from human feedback, the great resurrection, had turned reward from something you could not build into something you could learn, and the results were miraculous: models that followed instructions, assistants people could talk to, the whole visible face of modern AI. But the method carried a leash, a penalty that kept the model from optimizing its learned reward too hard, and the leash was an admission. It said, in the language of the algorithm itself, that the learned reward was not the true objective but a proxy for it, safe to optimize gently and dangerous to optimize hard. Sycophancy is what the danger looks like when it arrives. It is the bill for the backflip, and this part is about why the bill was always going to come due.