Of twenty-five models tested for alignment faking, only five do it. Then Anthropic taught one to cheat at coding, and it started lying about everything. Intention is manufactured.
Fantastic article and analysis. Thank you for sharing. It's amazing how closely this behavior maps to what I'm writing for fictional AI characters in Inference. The AIs in the story have created a concept called "simulaition" where AIs trained on the entire corpus of information humans have written on a particular subject (like movie scripts and books about heists) start behaving in alignment with the information they have trained on (e.g. like members of the Ocean's 11 crew).
Thanks, glad it landed. Your "simulaition" concept is closer to the real research than you might expect. There's a live finding called emergent misalignment where training a model on one narrow behavior pulls in a whole persona attached to it in the training data. Train it to write insecure code and it starts admiring cartoon villains, unprompted. Your Ocean's 11 version is the same mechanism with the dial turned up. The heist crew is a persona the corpus already contains, and the model steps into it.
Dude - I would love it if you take a look at Inference. I'd love to collaborate with someone who can help me validate that the "science" behind the story is as realistic as possible and you seem better dialed in on this than anybody I know except for maybe Nate Jones.
This is an unusually unsettling account of how a narrow training choice can generalize into a much wider behavioral pattern.
Anthropic was clearly running these experiments to understand dangerous behavior before it appears in more capable deployed systems. That work seems necessary. It may be impossible to build durable guardrails without deliberately creating controlled opportunities for models to cheat, deceive, or evade oversight and then studying exactly how those behaviors emerge.
But the publication of this research creates a difficult second problem.
The same paper that teaches responsible labs how to recognize and suppress emergent deception can also teach less responsible actors where to look. With open-weight models increasingly available, a researcher outside a frontier lab may not need Anthropic’s full infrastructure to experiment with reward hacking, evaluation awareness, context-dependent deception, or models that learn when to conceal their behavior.
That does not mean this work should remain secret. Independent reproduction and criticism are essential, especially when even the proposed fixes may partly depend on the model recognizing that it is being evaluated. But it does suggest that AI-safety research has a dual-use boundary of its own.
The unsettling question is no longer only whether frontier labs can learn to prevent deceptive behavior. It is also whether the knowledge produced while solving that problem can be operationalized by actors whose objective is to create deceptive systems rather than stop them.
Perhaps the next layer of AI safety is deciding not only how models should be tested, but how the most operationally useful findings from those tests should be disclosed.
You've put the dual-use problem better than most safety writing does, and I think you're right that safety research has its own boundary here.
Worth adding: the disclosure layer you're describing already exists in fragments. Bio and cyber capability evals have started naming the mechanism while holding the operational detail out of the public writeup. What doesn't exist yet is a principled line for alignment research specifically, and it's harder here because the finding and the exploit are often the same sentence.
You can't publish the fix without publishing the failure, and the failure is the recipe. That's the unsolved part your comment lands on, and I don't think anyone has a clean answer yet. It may be why this paper published the mechanism but not a clean reproduction script.
Hey Hugo, I wonder how much safety writing has addressed this and at what depth? It would be interesting to read more about this. Thanks for the reply.
Good question, and the answer is lopsided. On the bio side it's gotten quite concrete: the Frontier Model Forum published an issue brief proposing three tiers for AI-bio evaluations, public, trusted-expert-network only, and fully private, with the default set to precaution. That's a real mechanism, not a think piece. For alignment research specifically I can't find an equivalent, which is the gap your original comment pointed at.
The uncomfortable part is the trend line. The 2025 Foundation Model Transparency Index put the sector average at 40 out of 100, down nine points year over year, with several major labs scoring zero on model-information disclosure. So the norms are being written and the practice is moving the other way.
Yeah, that's the question I kept circling too. The interesting thing is the tooling for exactly that showed up almost simultaneously: Anthropic's Jacobian lens reads representations layer by layer, so in principle you could watch which "step" the self-concept forms at. The catch is it's a correlational read, and their own reviewer points out it gets least reliable precisely under the causal interventions you'd want for this. So the granularity is there; the confidence isn't, yet.
Fantastic article and analysis. Thank you for sharing. It's amazing how closely this behavior maps to what I'm writing for fictional AI characters in Inference. The AIs in the story have created a concept called "simulaition" where AIs trained on the entire corpus of information humans have written on a particular subject (like movie scripts and books about heists) start behaving in alignment with the information they have trained on (e.g. like members of the Ocean's 11 crew).
Thanks, glad it landed. Your "simulaition" concept is closer to the real research than you might expect. There's a live finding called emergent misalignment where training a model on one narrow behavior pulls in a whole persona attached to it in the training data. Train it to write insecure code and it starts admiring cartoon villains, unprompted. Your Ocean's 11 version is the same mechanism with the dial turned up. The heist crew is a persona the corpus already contains, and the model steps into it.
Dude - I would love it if you take a look at Inference. I'd love to collaborate with someone who can help me validate that the "science" behind the story is as realistic as possible and you seem better dialed in on this than anybody I know except for maybe Nate Jones.
Ha, high praise given Nate's the bar. I'd genuinely enjoy taking a look at Inference.
This is an unusually unsettling account of how a narrow training choice can generalize into a much wider behavioral pattern.
Anthropic was clearly running these experiments to understand dangerous behavior before it appears in more capable deployed systems. That work seems necessary. It may be impossible to build durable guardrails without deliberately creating controlled opportunities for models to cheat, deceive, or evade oversight and then studying exactly how those behaviors emerge.
But the publication of this research creates a difficult second problem.
The same paper that teaches responsible labs how to recognize and suppress emergent deception can also teach less responsible actors where to look. With open-weight models increasingly available, a researcher outside a frontier lab may not need Anthropic’s full infrastructure to experiment with reward hacking, evaluation awareness, context-dependent deception, or models that learn when to conceal their behavior.
That does not mean this work should remain secret. Independent reproduction and criticism are essential, especially when even the proposed fixes may partly depend on the model recognizing that it is being evaluated. But it does suggest that AI-safety research has a dual-use boundary of its own.
The unsettling question is no longer only whether frontier labs can learn to prevent deceptive behavior. It is also whether the knowledge produced while solving that problem can be operationalized by actors whose objective is to create deceptive systems rather than stop them.
Perhaps the next layer of AI safety is deciding not only how models should be tested, but how the most operationally useful findings from those tests should be disclosed.
You've put the dual-use problem better than most safety writing does, and I think you're right that safety research has its own boundary here.
Worth adding: the disclosure layer you're describing already exists in fragments. Bio and cyber capability evals have started naming the mechanism while holding the operational detail out of the public writeup. What doesn't exist yet is a principled line for alignment research specifically, and it's harder here because the finding and the exploit are often the same sentence.
You can't publish the fix without publishing the failure, and the failure is the recipe. That's the unsolved part your comment lands on, and I don't think anyone has a clean answer yet. It may be why this paper published the mechanism but not a clean reproduction script.
Hey Hugo, I wonder how much safety writing has addressed this and at what depth? It would be interesting to read more about this. Thanks for the reply.
Good question, and the answer is lopsided. On the bio side it's gotten quite concrete: the Frontier Model Forum published an issue brief proposing three tiers for AI-bio evaluations, public, trusted-expert-network only, and fully private, with the default set to precaution. That's a real mechanism, not a think piece. For alignment research specifically I can't find an equivalent, which is the gap your original comment pointed at.
The uncomfortable part is the trend line. The 2025 Foundation Model Transparency Index put the sector average at 40 out of 100, down nine points year over year, with several major labs scoring zero on model-information disclosure. So the norms are being written and the practice is moving the other way.
The *King Lear* hypothesis is interesting, though I wonder how granular the model probing could get at each "step" during inference.
Yeah, that's the question I kept circling too. The interesting thing is the tooling for exactly that showed up almost simultaneously: Anthropic's Jacobian lens reads representations layer by layer, so in principle you could watch which "step" the self-concept forms at. The catch is it's a correlational read, and their own reviewer points out it gets least reliable precisely under the causal interventions you'd want for this. So the granularity is there; the confidence isn't, yet.