An AI agent was handed a coding job and set about it with every sign of diligence. It said which file it would open, which command it would run and which check it would make. The trouble was that it only said so. The agent wrote its tool calls out as plain text, like an apprentice narrating a repair with both hands in his pockets, and nothing ran. The task scored 60 out of 100.
Then one file in the agent’s surroundings changed. It was a short piece of tool guidance that told it not to write pretend calls. The model was the same, its weights untouched, and the task was the same. The score went to 100.
The case comes from a paper posted on 9 October 2026 by researchers at Google Cloud AI Research and Northwestern University. Its results are the authors’ own and have not yet been independently replicated. The case is small and a little comic, but it points at something that matters every time you read that one AI model beats another. We picture two minds being compared. What was actually compared was two minds, each working inside its own set of instructions, tools and habits, with some amount of effort spent on each. A score belongs to all three of those things together, never to the model alone. Part of the work around the model can now be done by another AI, and on these tasks it moved enough to change which model came out ahead. So the question that tells you something about any claim of the form “X beats Y” is this: inside which system, and with how much help?
The overtake
The paper’s sharpest result is a reordering. On a benchmark of a hundred office-style tasks in code, web and documents, each scored out of 100 by hidden tests, file checks and model judges, the model GLM-5 averaged 64.2 in its standard surroundings. Google’s Gemini 3.5 Flash, in the same standard surroundings, averaged 72.4. By this benchmark’s own count Gemini is the stronger model, by about eight points.
After one round of editing its surroundings, GLM-5 averaged 75.7. That is 11.5 points higher than before, and 3.3 points above Gemini as Gemini came out of the box. Under a second off-the-shelf setup the pattern repeated: the edited GLM-5 scored 75.3 against Gemini’s 74.5. The authors put it plainly: a weaker model with an improved setup can outperform a stronger one in its original harness.
That sentence is true, and it is also narrower than it sounds. It does not say GLM-5 is better than Gemini. Give Gemini the same treatment and it rises too, to 77.5. What the result says is that the gap between these two models was smaller than the gap a round of editing could open, on these tasks. A ranking that size can be overturned by work done on neither model.
What surrounds a model
That surrounding layer has a name in the trade, the harness, and it is less mysterious than the word suggests. Think of what a new employee gets on the first day besides a brain: a written job description, a list of rules, notes from whoever held the job before, a binder of procedures, instructions for the office software, a few scripts that automate the tedious parts, colleagues to hand work to. A harness is the AI version of that kit. The paper splits it into eight kinds of component: the standing instructions, guardrails, memory, skills, tool guidance, scripts, helper agents and a small plug-in that can intercept what the agent does. Each one is a file. None of them changes the model. They change what the model is told, what it remembers and what it is allowed or reminded to do.
The analogy holds in one more respect. Put the same capable person in a well-run office and in a chaotic one and you will measure two different people. Nobody would call that a test of the person alone. Yet “model A scored 72” is usually read exactly that way.
How it improves without an answer key
The obvious way to improve a harness is to try edits and keep the ones that raise the score. That needs a score, and an agent working on a fresh task in the real world does not have one. Nobody has supplied the right answer. The paper’s method gets around this by comparing how the agent behaves rather than whether it is right.
One improvement step goes like this. The agent runs the task once with its current harness, and that run becomes the reference. An editor model reads what happened and proposes two edited harnesses. One edit targets general weaknesses in how agents work, such as planning, tool use and keeping track of context. The other targets the knowledge and rules of this particular domain. Both edited versions are then run on the same task, and their runs are compared with the reference. Did the edit fix the behaviour it aimed at, change nothing, or break something else? A final step keeps the edits the runs support, merges them into one harness and runs that to produce the deliverable. Afterwards the system writes down what it learned in a library that later tasks can consult.
The method’s own description is candid about what the reference run is for: it “reveals how the harness behaves, not whether its deliverable is correct.” That is the honest centre of the method. It cannot tell a right answer from a wrong one. What it can see is behaviour, and the behaviour it fixed turned out to be the kind you can see.
Look at the failures it repaired. In one professional task, an agent had to produce a memo from several source files: schedules, venues, labour, budgets. The first run scored 43.1. Its record showed the agent never reconciled figures across the files and never reopened what it had written to check it. The edits forced it to list every input, trace each number to its source and run a script that reopens the finished files. The score rose to 77.8. In a legal task, the agent had to mark up a procedural order in an arbitration. It put its changes in Word’s track-changes layer, so anyone reading the file as plain text saw only the accepted wording, and the markup was invisible. A new skill made it write insertions and deletions into the body of the text. The score went from 44.4 to 94.5. In a third case, a research assistant kept researching and, near the deadline, still had no files. A plug-in that warns the agent after a fixed number of tool calls with nothing written fixed it.
None of these is a gap in knowledge. The agents knew enough law, enough arithmetic and enough code. What they lacked were habits: check your work, write what you found, make sure the reader can see your changes, actually press the button. Habits are visible in a transcript in a way correctness is not, and that is why comparing runs can find them without an answer key. It may also be why, on two of the three benchmarks, the weaker model gained more. I would guess it simply had more bad habits to lose, though the authors’ handful of hand-picked cases cannot settle that.
What the help cost
The edited GLM-5 did not win alone. Its harness was rewritten by Claude Opus 4.8, a stronger model than either contestant, and each task was run several times before the final deliverable: once for the reference, once for each candidate, once for the merged version. Gemini, in the comparison, ran once in its standard harness. The overtake is GLM-5 plus a stronger editor plus extra runs, against Gemini with none of that. The costs were not equal, and the result does not claim they were.
Two pieces of evidence keep this from collapsing into “a stronger model did the work.” First, the authors ran a version in which the same model edited its own harness, across five models ranging from GLM-5 to Claude Opus 5. Every one improved on every benchmark in that test, so the method does not depend on a stronger editor, though the main tables still use one. Second, by the authors’ own accounting, which counts every token spent on editing as well as on doing the task, their method gave the best score for the compute among the approaches compared, over the range of budgets they tried. That is a claim about a bounded range, not proof that this is the best way to spend the money.
Nor does editing a harness help by default. Two other published methods for improving agents sometimes scored below the untouched harness, one by as much as 6.6 points on coding tasks. The other fell below the starting harness in seven of twelve benchmark and setup pairings. When the paper’s own editor tried rewriting the harness directly, without reference runs or the library to compare against, coding scores fell from 73.4 to 63.2. The layer around a model is large enough to move a score a long way in either direction. What moved it upward here was the discipline of checking each edit against what the agent actually did.
How far the evidence reaches
My view, at the strength these results support, is this. On office, professional and legal tasks, the layer around the model is large enough to reorder models and cheap enough to move per task. That much the paper shows. It does not show that the improvements are correct in any deeper sense. The work was graded partly by model judges, and on the legal benchmark the score counts rubric points passed rather than whole tasks done right. The authors themselves say that comparing behaviour, without ground truth, cannot guarantee correctness or show that a change will carry over to a different task. The test sets are small, from 25 to 100 tasks per benchmark domain, so a few tasks can swing an average. And nothing here shows an improvement that keeps compounding: the authors did not test very long streams of tasks, and they do not claim unbounded self-improvement.
Even with all that, the direction seems clear to me. If a model’s surroundings can be edited by an AI, on the task, in the time it takes to run the task a few times, then “the model in its default harness” stops being the natural unit of comparison. It becomes one configuration among many, and often not the one anyone will actually use. I do not expect a better model to make the layer disappear. The fixes here were about the particular environment, such as a layer of markup the reader never saw, or a research task nearing its deadline with its files still unwritten, and every new workplace brings its own. Even the strongest model in the paper gained a few points from its own edits. A better model may need fewer reminders. It will still be working inside some office, and someone or something will have written its binder.
The agent that narrated its tool calls was not a lesser mind than the one that ran them. It was the same mind, with one sentence missing from its instructions. The next time a chart says one model beats another, the chart is reporting on two minds in two offices, after a certain number of tries. Until you know who furnished those offices, and how many tries each model got, you know less about the minds than the chart suggests.
This is Episode 17 of Reading the Frontier. What the frontier’s evidence shows, and what we think it means.