The Attention Wager, Part 5: The Neglected Cast
Three sentences for two thirds of the parameters. The component with no name turned out to hold what the model knows. Memory that is not attention, and never was.
This is The Attention Wager, a close reading of Attention Is All You Need, one section and one bet at a time.
In 1972 Endel Tulving proposed that long-term memory is not one thing, and he did it in a book chapter rather than an experiment, because what he was disputing was not a result but a habit.
There is a store of events, tied to a time and a place, which is what you use when you recall where you were when you heard some piece of news. And there is a store of general knowledge, unattached to any occasion of learning: that Paris is in France, that promises are usually kept, that the word “bank” has two meanings. You know the second kind of thing without remembering acquiring it. Tulving called them episodic and semantic memory.
The part of his argument that gets lost in summary is that he did not think they were separate boxes. From 1972 onward he held that the two are interdependent, and that their interaction is a necessary feature of memory working normally. Retrieving an event depends on what you know in general; what you know in general was assembled out of events.
Four articles of this series have now been about a mechanism for deciding what in the present moment is relevant to what else. That is only half of what a mind needs. Somewhere there also has to be a store of what is generally true, and it cannot be the same organ, because the first one holds nothing when the room is empty.
Ask a transformer what the capital of France is and attention will do its work on those seven words, relating each to the others, and none of them contains the answer. The answer is not in the sentence. It has to come from somewhere that is not the sentence.
The transformer has that somewhere. It is in Section 3.3, it got three sentences, and it does not have a name.
