The Attention Wager, Part 4: Where Order Comes From
Two options, tested, tied. The tie was broken by a prediction that failed. Nine years later the question is still open, and the oldest version of it was posed with a pun.
This is The Attention Wager, a close reading of Attention Is All You Need, one section and one bet at a time.
In September 1948, at a symposium at Caltech attended by John von Neumann among others, Karl Lashley read a paper on why behaviour cannot be a chain.
The prevailing account said it could. A sequence of actions was held to work like falling dominoes: the sensation produced by movement n is the trigger for movement n+1, whose sensation triggers n+2, and so on down the line. Lashley thought this was wrong, and he had evidence. Movements continue when sensory feedback is cut. Some sequences run faster than the feedback loop that supposedly drives them. A pianist’s fingers do not have time to feel their way. He argued that order has to be represented somewhere other than in the elements themselves, as a plan that exists before the sequence begins.
To show why order could not live inside the elements, he built a sentence.
The mill-wright on my right thinks it right that some conventional rite should symbolize the right of every man to write as he pleases.
Six occurrences of the same sound. Six different words. Nothing in the sound distinguishes them. What distinguishes them is where each one falls, and a listener who could not tell where would hear a sentence with a hole in it six times over.
Lashley’s paper appeared in print in 1951, three years after he delivered it. He was arguing about muscles and speech, not about matrix multiplication, and it would be an overreach to say he anticipated anything about neural networks. But he had specified a problem precisely, and the specification outlived the argument it was built for. If your elements are identical and their function is not, then something outside the elements has to carry the order.
Sixty-six years later, a mechanism was proposed that could not see order at all.
