Robonaissance

Robonaissance

Whatever You Ask For, Part 5: Your Organization Was Reward Hacking Before the Machines Arrived

A sales target that invented three million accounts. A law nobody wrote. A clock that stops at the hospital door. Machines did not bring this problem. They are removing the brakes.

Hugo's avatar
Hugo
Jul 25, 2026
∙ Paid

This is Part 5 of Whatever You Ask For, a series on machines that grant wishes exactly as worded, and the wording that remains ours.


Forty-Two a Day

Julie Miller managed a Wells Fargo branch in Allentown, Pennsylvania. The target she was given, as she later described it to a reporter, was seven checking accounts and forty-two products a day. Her account of how the branch met it is not a story about fraud. It is a story about asking. They begged customers to open accounts, she said, so that they would not lose their jobs.

The number came from a strategy the bank had discussed publicly for years. Wells Fargo measured itself on cross-selling, the number of separate financial products held by each household, and it made no secret of wanting that number to reach eight. The slogan inside the company was the Great 8. Executives cited the cross-sell ratio in quarterly results, where it hovered around six. Analysts admired it. It was, by any conventional standard, a well-communicated, well-understood, universally visible objective.

Below the slogan sat the machinery. According to accounts from former employees, branch managers were expected not merely to hit the daily targets handed down from regional bosses but to exceed them by a fifth, and those who fell short could expect to be named on a daily conference call, in front of every other manager on the line.

Consider what that arrangement looks like from a branch in Allentown on a Tuesday. The day has a number attached to it. The number does not adjust for how many people walked through the door, or how many of them needed anything. There is no mechanism for reporting that the target is not achievable in this location this week, because the target is not a forecast, it is an expectation. And at the end of the day there is a call. The people who make the number are not on it in any meaningful sense. The people who miss it are.

What happened next is a matter of regulatory record. On the eighth of September 2016, Wells Fargo was fined one hundred and eighty-five million dollars, split between the Consumer Financial Protection Bureau, the Office of the Comptroller of the Currency, and the city and county of Los Angeles. It was the largest penalty the CFPB had imposed on a financial institution. The bureau found that employees had opened more than two million deposit and credit card accounts that customers may not have authorized, roughly one and a half million of the former and over half a million of the latter. Money was moved between accounts without permission. Debit cards were issued and PINs created. In some cases employees invented email addresses so that customers could be enrolled in online banking without knowing. Around five thousand three hundred employees were dismissed, a figure that comes from the Los Angeles City Attorney’s office.

A year later the bank widened the review window, looking at January 2009 through September 2016 rather than the shorter period first examined, and found roughly one and a half million more, bringing the total of potentially unauthorized accounts to around three and a half million. The two figures are not a revision of the same count. They are two different windows, and the larger one is larger partly because it is longer.

The convenient reading is that a bank had a bad culture and some dishonest staff. That reading is comfortable and it explains nothing, because the same shape appears in hospitals, schools, police forces, factories and research universities, in countries with entirely different cultures, legal systems and labour markets, run by people with no connection to one another.

Something more general is going on, and it was named a long time ago. Or rather, it was named several times, by several people, in a sequence with its own comic aptness.


The Law That Lost Its Own Attribution

Almost everyone who works with metrics knows the sentence. When a measure becomes a target, it ceases to be a good measure.

It is called Goodhart’s law. Charles Goodhart did not write it.

What Goodhart wrote, in a 1975 paper delivered at a Reserve Bank of Australia conference and concerned with British monetary policy, was narrower and more technical: that any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes. He was describing something specific. The Bank of England had noticed dependable relationships between certain measures of the money supply and inflation, and when it began steering by those measures the relationships fell apart, because banks and markets adjusted to the new rules and the correlation that had held under the old ones stopped holding.

Meanwhile Donald Campbell, a social psychologist working on research methodology and educational testing, had arrived at a broader version, and had arrived at it earlier. Jeffery Rodamar, writing in Significance in 2018, traced Campbell’s formulations back to 1969, with publication in 1975 drawing on conference presentations from the previous year. Campbell’s version is the one that reads like a warning: the more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures, and the more apt it will be to distort and corrupt the social processes it is intended to monitor.

The famous sentence appears in neither. It comes from Keith Hoskin, writing in 1996, who attached Goodhart’s name to the general principle and gave it its compact form, observing that every measure which becomes a target becomes a bad measure, and that this was being recognized, ruefully, as one of the governing laws of the age. He added an argument that is easy to skip past and worth stopping on: that this is the inevitable corollary of accountability itself, an invention of modernity rather than a defect in it.

Marilyn Strathern, an anthropologist writing the following year about audit culture in British universities, quoted Hoskin’s formulation in a paper on how universities were being rated. The sentence, and eventually the credit for it, attached itself to her.

So the most quoted line in the criticism of measurement is credited to an economist who did not write it, popularized by an anthropologist who was quoting someone else, while a psychologist who got there first is filed under a different name in a different literature. A law about how things distort under measurement pressure lost track of its own attribution.

The tangle is funnier than it is important, but there is something important underneath it. Four people, working on monetary policy, educational testing, accounting history and university audit, none of them reading each other, arrived at the same principle within about thirty years. That is what independent discovery looks like, and it is the signature of a structural property rather than a local pathology. This is not an observation about economics. It is a property of any system in which behaviour is steered by a number.

User's avatar

Continue reading this post for free, courtesy of Hugo.

Or purchase a paid subscription.
© 2026 Robonaissance · Privacy ∙ Terms ∙ Collection notice
Start your SubstackGet the app
Substack is the home for great culture