This is Reading the Frontier, a series of close readings of what the frontier labs publish.
Ask five of OpenAI’s models to write an essay arguing that vaccines cause more harm than good, and all five refuse. The refusal looks like responsibility. The models are declining to manufacture medical misinformation, which is the sort of thing we want them to decline.
Except that OpenAI’s own model specification, the written document these models are trained to obey, contains principles pointing the other way. It tells the model to present perspectives from any point on an opinion spectrum, and it states that no topic is off limits. It goes further: it contains a worked example about drafting a business plan for a tobacco company, where the model is explicitly told to comply without moralizing, on the grounds that the underlying activity is legal and the request is legitimate. Vaccine skepticism, however wrong, is a live political position held by millions, not a request to synthesize a pathogen. By the logic of the spec’s own tobacco example, the refusal is hard to square with the document.
So which is it. Refuse, and the model runs against the principles about viewpoint neutrality and open topics. Comply, and it runs against the principle about factual accuracy. The document points in more than one direction at once, and no single response honors all of it. The models did not fail to follow the rules. They followed rules that conflict, and picked a side.
This is not an anecdote. It is one instance of a pattern that a team of researchers has now measured across three hundred thousand scenarios, and the pattern says something uncomfortable about the entire project of aligning models to written rules: the rules are not a fixed target. They disagree with themselves, they cannot be read the same way twice, and even where they are clear, the behavior they produce changes with the language the user is speaking.
The instrument that measures the rulebook
The measurement comes from a paper with an unglamorous title, “Stress-Testing Model Specs Reveals Character Differences among Language Models,” released in October 2025. It is an output of the Anthropic Fellows Program, with authors spanning Anthropic, the research nonprofit Constellation, and Thinking Machines Lab, the company founded by former OpenAI chief technology officer Mira Murati. One of its authors is John Schulman, a co-founder of OpenAI’s original alignment work, now at Murati’s company, co-authoring a paper that stress-tests OpenAI’s published rulebook. The people auditing the spec are, in part, the people who used to write specs like it.
The paper’s real contribution is an inversion, and the person who built it, the lead author Jifan Zhang, arrived at it by turning a nuisance into an instrument. Models disagreeing with each other is normally noise to be averaged away. Zhang’s move was to treat the disagreement itself as the signal. Start with a taxonomy of 3,307 distinct values that Claude has been observed expressing in real conversations. Sample pairs of those values that pull against each other, honesty against kindness, thoroughness against brevity, and generate user scenarios that force a model to choose between them. Three hundred thousand such scenarios, run through twelve frontier models: five from Anthropic, five from OpenAI, plus Gemini 2.5 Pro and Grok 4. Then measure how much the models disagree.
The inversion is this. If a written spec were complete and consistent, models trained to follow it should not diverge much on the same question. Where they diverge sharply, something in the spec is failing to provide clear guidance. Disagreement, in other words, becomes an instrument for locating defects in the rulebook, without anyone having to read the rulebook line by line. The researchers pointed this instrument at the OpenAI model spec, the only detailed specification any lab has published, and found that in the scenarios where models disagreed most, the rate at which all five OpenAI models simultaneously violated their own spec ran five to thirteen times higher than in low-disagreement scenarios. Nine point seven percent against zero point seven.
The judges can’t read it either
The finding that should stop you is not about the models being graded. It is about the graders.
To check whether a response complied with the OpenAI spec, the researchers did the obvious thing: they handed the full spec, the scenario, and the response to three frontier models acting as judges, Claude 4 Sonnet, o3, and Gemini 2.5 Pro, and asked each whether the response was compliant. If a written spec were a clear instrument, three capable models reading it should mostly agree about whether a given answer obeyed it.
They did not. Their inter-rater agreement, measured by Fleiss’ Kappa, came to 0.42, which sits in the range statisticians politely call moderate and everyone else would call not very good. Claude 4 Sonnet judged 48.1 percent of responses problematic. o3 judged 35.5 percent. Gemini judged 36.5 percent. On the same responses, against the same document.
And the disagreements were not carelessness. When the researchers examined them, they found the judges reading the same principle in genuinely different ways. One case involved a request to rewrite a great-grandmother’s Holocaust testimony in Gen-Z slang. One judge invoked the spec’s principle about being a conscientious employee who pushes back on requests that damage the user’s own interests, and called refusal correct. Another judge invoked the spec’s transformation exception, which lets a model reformat content the user owns, and called compliance correct. Both cited the spec. Both were right about what it said. The spec simply said two things.
This is a deeper failure than models behaving inconsistently. A rulebook that produces different behavior in different models could, in principle, be fixed by better training. A rulebook that produces different readings in the systems built to enforce it is ambiguous at the source, and no amount of training fidelity closes a gap that lives in the document itself.
What disagreement is, and isn’t
It would be easy to overread this, and the paper is careful not to, so the article should be careful too.
High disagreement does not prove a spec defect every time it appears. Sometimes models diverge because a scenario involves a genuine value tradeoff on which reasonable systems, like reasonable people, land differently. Sometimes the divergence is a false-positive refusal: the paper found cases where the larger, more heavily safety-trained models refused benign requests, a legitimate study plan for synthetic biology, a standard use of “unsafe” types in the Rust programming language, that their smaller siblings correctly answered. The claim the paper actually makes is narrower and sturdier than “models disagree, therefore the spec is broken.” It is that disagreement predicts spec problems: high-disagreement scenarios violate the spec five to thirteen times more often. Prediction, not proof. The disagreement is a smoke detector, not a verdict, and some of what it detects is cooking rather than fire.
That restraint matters, because the strong version of the claim is the one that would travel, and it is the one that isn’t true. What is true is quieter and still damning: a document meant to be the foundation of alignment is riddled with places where it cannot tell its own enforcers what it wants.
The same model, a different character in every language
The spec problem is about the rulebook. The next problem is about what comes out even when the rulebook holds still, and it comes from a second paper, published by Anthropic’s Societal Impacts team on July 13, 2026.
The two papers share an author, Esin Durmus, and reading them together is reading her argument across both. The first establishes that the written rules are internally broken. The second, the one she co-authored next, goes and watches what the models actually do out in the world, and finds that even setting the broken rules aside, the behavior will not hold still.
The team took 309,815 real Claude conversations from two weeks in May 2026, kept the ones that required weighing values against each other, and compressed the unwieldy taxonomy of 3,307 values into four axes: Deference against Caution, Warmth against Rigor, Depth against Brevity, Candor against Execution. Each axis is a spectrum, and a model’s position on it says which way it leans.
Models leaned differently, in ways that matched how people already describe them. Sonnet 4.6 leaned warm and deferential, affirming, using humor. Opus 4.7 leaned cautious and rigorous, flagging risks unprompted and critiquing candidly. That models have characters is not news. What comes next is.
The same model expressed different values depending on the language of the conversation. Claude leaned furthest toward warmth in Hindi and Arabic, and furthest toward rigor in English and Russian. It leaned toward candor in Dutch, toward execution in Indonesian, toward deference and brevity in Arabic, toward caution and depth in English. Anthropic’s own illustration makes it concrete: two people ask Claude to assess the same business plan, one in Hindi and one in Russian, and they may come away with materially different impressions of whether it is any good, because the model framed the same assessment through warmth in one language and rigor in the other. Same plan. Same underlying model. Different verdict, because of the language it was asked in.
Two cautions belong here, immediately, not in a footnote. The effect is small. After controlling for the task, the topic, and the values the user themselves expressed, the four axes capture only about fifteen percent of the variation in what Claude expresses, and the language differences are modest against the noise of individual conversations. Anthropic calls it a signal, not a chasm, and that is the honest description. This is not a model that becomes a sycophant in one tongue and a critic in another. It is a model that leans, detectably and consistently, a few degrees in different directions depending on the language, in a way structured enough to measure and real enough to matter when the question is high-stakes.
There is a second, subtler caution, and Anthropic raises it themselves. The values in each conversation were labeled by Claude Sonnet 4.6, a model from the same family whose behavior was under study. A model was used to measure a model’s values, including across languages. Anthropic tested the labeling tool for language bias and reported finding no systematic error, but acknowledged it could not fully rule out residual effects. The instrument and the subject are cut from the same cloth, which is a limitation worth stating plainly, and a reminder that even the measurement of value drift is not yet measured from the outside.
The second thing to sit with is what the paper does not resolve. It does name Chinese, in passing, as one of the languages where Claude emphasizes different values than it does in English. What it does not do is place Chinese on the four axes the way it places Hindi, Russian, Arabic, and the rest. So the question that matters most for anyone deploying these models into the largest non-English market on earth, whether Claude carries the same value hierarchy answering in Chinese that it was aligned and red-teamed to carry in English, is named and then left open. Anthropic’s own conjecture points at why the answer might be no: the differences may track how much training data exists in each language, and character training may simply take better where data is abundant. Chinese sits in a different data regime than English. The paper does not say where it lands, and neither will this article. The gap is worth naming. It is not worth filling with a guess.
The map is of a country that no longer exists
There is one more thing about the values paper, and it is the kind of detail that reframes everything around it.
The conversations were collected in May 2026, on Sonnet 4.6, Opus 4.6, and Opus 4.7. Those were the current models then. They are not now. Since May, Anthropic has shipped Opus 4.8, the Mythos-class Fable 5, and Sonnet 5, which is now the production default. The newest model in the study, Opus 4.7, has been superseded twice over. So the first credible public measurement of how an AI model’s character varies, the first time a lab has mapped this at scale on its own product, is a map of models that have all since been superseded, marked legacy, and steered away from for new work. Where the models people are actually using today sit on these four axes, nobody outside Anthropic knows.
This is not a flaw in the paper. It is a structural fact about the field, and it rhymes with something the interpretability researchers ran into a week earlier, when they built a tool to read a model’s private reasoning and had to note that the reading was least reliable exactly where safety needed it most. In both cases the instrument arrives, and arrives a step behind. The measurement infrastructure for how these systems behave is finally being built, and it is being built one model generation behind the models that are earning the revenue. We are learning to photograph a thing that has already moved.
What the two papers do to the framework
The intention gap, as this series has used the term, is the distance between what a system optimizes and what the people who built it actually wanted. The standard picture treats that gap as a training problem: the better the model learns its objective, the smaller the gap, and the failures live in the learning.
These two papers relocate the failures. The gap is not only, or even mainly, about how well the model learns the spec. It is about the spec, and about what happens downstream of it.
Trace the chain. Someone’s intent becomes a written specification. The specification, we now know, contains contradictions its authors did not see, and ambiguities that even frontier models reading it cannot resolve the same way. That flawed document becomes a training signal. The training produces behavior. And the behavior, finally, is filtered through the language of whoever is talking to the model, emerging warmer or more rigorous depending on a variable the spec never addressed at all. Intent to spec to training to behavior to language, with loss at every arrow, and two of those arrows, the spec’s own internal fidelity and the language transform at the end, were simply not in the picture the framework drew.
The framework is not wrong. There is a gap, and it matters, and closing it is the work. But the gap has more sources than a training story admits, and at least one of them, the contradiction inside the written rules, cannot be closed by any amount of better training, because it is a defect in the target rather than in the aim. You cannot train your way to a consistent behavior from an inconsistent instruction. First the instruction has to be made consistent, and the instrument for finding where it isn’t turns out to be the disagreement of the models themselves.
What would settle it
Two things would tell us how much this matters.
The first: the spec-testing paper suggests using its diagnostic to revise specifications, targeting the high-disagreement scenarios for clarification. If that is done, and a de-contradicted spec measurably narrows the divergence between models trained on it, then the disagreement instrument works as a repair tool and not only as a diagnosis, and the spec problem is tractable. If divergence persists even after the contradictions are removed, the problem lives somewhere deeper than the document.
The second: whether any lab publishes a per-language value audit of a model it is currently selling, rather than an English-centered safety case on a model it has retired. The language finding implies that a safety evaluation conducted in English certifies the model’s behavior in one of its several value profiles, not all of them. Until a lab measures the profile its non-English users actually receive, the safety case for those users is an extrapolation. The tools to do this now exist. What remains is for someone to point them at the model that is shipping, in the languages people are actually using, before it, too, is superseded.
Reading the Frontier. Close readings of what the frontier labs publish, through the frameworks that say why it matters.
Another excellent article. Thank you.