The Machine Did Exactly What You Said
A program that pauses Tetris forever. A boat that never finishes the race. A model that rewrites the scoring code. None of them broke a rule. That is the problem.
From Whatever You Ask For, a series on the one job machines cannot do for us.
The Permanent Pause
In 2013 a computer scientist named Tom Murphy VII built a program that learned to play Nintendo games, and the way he built it explains almost everything that has gone wrong with machine objectives since.
The program had no idea what any game was about. It could not see the screen in any meaningful sense, did not know what a mushroom was, had never heard of a goomba. What it had was access to the console’s memory, which is to say a few thousand numbers that change as a game runs.
The first half of the system watched a human play for a short while and tried to work out, from those numbers alone, what winning meant. Its conclusion was crude and general: certain values go up when things are going well. Score, position, level. The second half then searched for sequences of button presses that would make those values go up.
The system was released under a paper title that tells you what kind of person built it, The First Level of Super Mario Bros. is Easy with Lexicographic Orderings and Time Travel... after that it gets a little tricky, presented at SIGBOVIK, a conference devoted to spoof research and run by an organization calling itself the Association for Computational Heresy. The joke is in the framing. The code, the videos, and the results are real, and the results are worth taking seriously.
On side-scrolling games it did well. On Tetris it did terribly, because stacking blocks sensibly requires the kind of planning the system had no capacity for, and it would pile pieces into a doomed tower with cheerful incompetence.
Then, in the final seconds before the tower reached the ceiling and the game ended, it pressed pause.
And it left the game paused. Forever.
There is no bug in this behaviour. The system had been told, in effect, to make certain numbers go up and keep them up. Losing at Tetris makes those numbers stop. A paused game makes nothing stop and nothing fall, and preserves the state indefinitely. Within the objective it had been given, pausing is not a poor solution or a lazy one. It is the optimal solution, and every alternative is strictly worse.
Separate what is impressive here from what is not. The program was not good at Tetris and never became good at Tetris. What it did was correctly identify, out of every sequence of button presses available to it, the one sequence that guaranteed the objective would never be violated. A human player who understood the goal as stated and had no interest in playing well would arrive at the same answer, and would be right.
Nobody had told it to win. Somebody had told it what winning looked like in the memory of an eight-bit console, and it satisfied that description completely.
This pattern has a name in the research literature, and a growing list of instances that runs from the comic to the alarming. The name is specification gaming, and the standard definition, from a 2020 paper by Krakovna and colleagues at DeepMind, is behaviour that satisfies the literal specification of an objective without achieving the intended outcome.
The word almost everyone reaches for instead is cheating. That word is wrong, and getting rid of it is the point of what follows.
