Whatever You Ask For, Part 8: The Genie Never Picks the Wish
The same method predicts protein structures and wins at chess. It has no view on which is worth doing. That gap does not close as machines improve. It is not a gap in the machines.
This is Part 8 of Whatever You Ask For, a series on machines that grant wishes exactly as worded, and the wording that remains ours.
Two Axes
In 1994, a biologist named John Moult set up an examination, because everyone in his field was claiming progress on protein structure prediction and nobody could check. Every two years after that, sequences went out to the world, predictions came back, and they were scored against structures determined in laboratories. For most of the next twenty-six years the best scores sat around sixty on a scale of a hundred. He had built, and kept running, an assessment that mostly documented failure.
In 2020 he stood in front of his audience, walked them through the history of the competition, and then showed the graph. One line stood where no line had ever been. The problem, for single protein chains, was solved.
Within roughly the same window, a closely related recipe of large networks and enormous computation, pointed at a different target, learned chess, shogi and Go from nothing but the rules and games against itself, and became stronger at all three than any human and any program humans had built.
One of those targets was a scientific problem of the first rank, with consequences running through drug design and the understanding of disease. The other was a board game.
The recipe made no distinction. Nothing in the method knows or could know that curing disease matters more than winning at Go, and no amount of additional capability would produce that knowledge. Scale it up a thousandfold and it gets better at hitting the target. It does not begin to have views about which targets deserve hitting.
That is the shape of the problem. Capability is one axis. What is worth pursuing is a different axis. The two do not run parallel, and progress along the first does not carry you anywhere along the second.
Most people accept this readily about machines and then quietly assume it stops being true at some threshold of sophistication. If a system were truly intelligent, the intuition goes, wouldn’t it work out what actually matters? Wouldn’t we, if we were smart enough? A great deal rests on that intuition, including the widespread comfort that as our systems get more capable, the problem of telling them what to want will somehow take care of itself.
The intuition has an obvious test case, and we are it. Human beings have gotten remarkably better at the first axis over a few thousand years. We know vastly more about how the world works than anyone did in 1500, and we can do things with that knowledge that would have looked like sorcery. Set that against how much closer we have come to agreement about what is worth doing with a life, or what a society owes its weakest members, or how to weigh the people alive now against the people who will be alive in two hundred years. The first curve is close to vertical. The second one is not obviously a curve.
That is not proof of anything, since a determined optimist can say we simply have not gotten smart enough yet. But it is a strange record for a thesis that says capability drags value understanding along behind it.
It will not, and the reason has nothing to do with the current state of the technology.
Why Getting Smarter Does Not Hand You a Goal
Start with something small enough to see all of.
A team is choosing between two suppliers. One quotes a lower price. Someone says: their bid is thirty per cent cheaper, so we should go with them. Everyone nods, and the meeting moves on.
Now look at what happened in that sentence. The first half is a fact, and it could be checked. The second half is not a fact of the same kind, and no amount of checking the first half produces it. To get from the price to the decision, something else had to be true: that lower cost is what we are trying to achieve here, or at least that it outweighs everything else in play. Nobody said that. It was carried into the room and never set down where anyone could look at it.
This is not a trick of that particular sentence. It happens every time reasoning arrives at what should be done. Pile up facts as high as you like, about costs, consequences, probabilities, and side effects, and the pile does not tip over into a conclusion about what to do. Somewhere in the chain there is a premise that says one of these outcomes is better than another, and that premise is doing the decisive work while attracting the least attention.
The observation is usually traced to a passage David Hume wrote in the third volume of his Treatise of Human Nature, published in 1740. He noticed that authors writing about morality proceed for a while in the ordinary way, making claims about how things are, and then without warning every sentence has an ought in it. The change is easy to miss, he said, and it matters enormously, because this new relation needs explaining, and it is hard to see how it could be deduced from the entirely different relations that came before.
Honesty requires adding that scholars have argued for a century about what exactly Hume meant, and that the strong version, sometimes called Hume’s Law, is a reading later writers attached to the passage rather than something he demonstrated. He says other things nearby that sit awkwardly with it. So the passage is not a proof. It is a place to put your attention, and once your attention is there the pattern is hard to unsee.
The same structure holds for a machine, and holds more starkly, because a machine’s premises are written down.
Imagine a system with perfect and complete knowledge of consequences. Give it any action and it tells you, with certainty, everything that would follow: who is affected, how, for how long, at what cost, with what second-order effects a century out. This is far beyond anything that exists and it is worth granting anyway, because it isolates the question. Such a system still cannot tell you what to do. It can tell you that one path leads to more people alive and another to more people wealthy. Which of those descriptions is the better outcome is not among the consequences it enumerated. That judgment has to be supplied, and supplying it is a different kind of act from computing.
The version of this claim that circulates in AI research is Nick Bostrom’s orthogonality thesis, from a 2012 paper. He puts it carefully, and the care is easy to strip out in repetition. Intelligence and final goals, he writes, are orthogonal axes along which agents can vary; more or less any level of intelligence could in principle be combined with more or less any final goal. The hedges are deliberate. He also notes that the thesis says these combinations are logically possible, not that it would be practically easy to build a superintelligent system holding some particular goal.
And there is a feature of the argument that its critics press and that anyone using it should state plainly. Bostrom is using intelligence to mean something specific: skill at means-end reasoning, the ability to figure out how to get what you are after. Given that definition, the thesis is close to true by construction, since means-end skill is defined in a way that says nothing about ends. Critics have argued that this is not what we mean by intelligence in the fullest sense, and that a sufficiently general intelligence might not be separable from wisdom about ends in the way the thesis assumes. That is a live argument, not a settled one.
Which is why the weight here does not rest on the thesis. It rests on the supplier meeting. Watch any chain of reasoning that ends in a decision, and find the step where something got called better. That step is not a measurement. It never was.
