The Attention Wager, Part 3: Many Heads
Eight heads, one sentence of justification, and five years before anyone could say why it worked. The mechanism that binds separate features into a single object.
This is The Attention Wager, a close reading of Attention Is All You Need, one section and one bet at a time.
Section 3.2.2 of “Attention Is All You Need” is about a page long, and most of that page is notation.
It describes a modification to the mechanism Part 2 took apart. Rather than run one attention function over full-dimensional queries, keys, and values, project them down eight times with eight different learned linear maps, run attention in each of those lower-dimensional spaces in parallel, concatenate the eight results, and project once more. Eight heads. Sixty-four dimensions each, because the model width is 512 and 512 divided by 8 is 64.
Then comes the justification, and it is one sentence. Multi-head attention, the authors write, allows the model to jointly attend to information from different representation subspaces at different positions, and with a single head, averaging inhibits this.
That is the whole theoretical case. One sentence about subspaces. It is followed by a second sentence noting that because each head works in a reduced dimension, the total cost is about the same as one full-dimensional head, which is an argument about the budget rather than about the idea.
Every frontier language model in the world today has this component in every layer. In 2017 it was justified in twenty-nine words.
The paper’s credit footnote says whose twenty-nine words they were. Noam Shazeer proposed scaled dot-product attention, multi-head attention, and the parameter-free position representation, which is to say the subject of this article and the subjects of the two on either side of it. Part 1 met him walking down a corridor, overhearing a conversation he had not been invited to. Part 2 found him supplying the denominator that kept the mechanism trainable at scale. Here he supplies the component that would occupy the interpretability field for a decade, and the paper gives his reasoning one sentence.
There is no record of why he chose eight. The footnote says what he proposed. It does not say what he was thinking, and neither does anything else in the public record.
What follows is the story of how the field spent five years finding out what those twenty-nine words had actually described, and discovering that the answer was something else.
