This is Episode 7 of The Architecture of Memory: how intelligence reaches across time when it cannot carry the whole past with it.
“The rate can always be reduced by throwing away details of the original signal x.” Naftali Tishby, Fernando Pereira and William Bialek put that sentence near the start of a paper they dated 30 September 1999 and posted to arXiv the following year. It reads like a complaint about a loophole, and it is one. Information theory calls the number of bits a compressed record spends on each symbol of the original its rate. A lower rate means a smaller record, a cheaper message, a lighter store. Suppose you hand a compression scheme the original and ask only for a lower rate. It can always comply, because every detail it drops makes the record cheaper. The cheapest record of all keeps nothing, and by rate alone it is perfect.
So “make it smaller” is not a complete instruction. A scheme also needs to be told which details matter, so that it drops the others. The question is where that instruction comes from. Three formal treatments, two papers by Claude Shannon and the one above, handle it the same way. The scheme does not work out what matters by itself. The rule arrives with the problem, written by someone outside the scheme, and the scheme’s job is to obey that rule as cheaply as possible.
A scope, not a verdict
Claude Shannon’s 1948 paper, “A Mathematical Theory of Communication”, says early on what it will leave alone. Messages often have meaning, he grants, and then: “These semantic aspects of communication are irrelevant to the engineering problem.” The sentence is easy to misread. It does not say meaning is unimportant, and it does not offer a theory of meaning. It marks the edge of a problem. The next sentence says what lies inside: “The significant aspect is that the actual message is one selected from a set of possible messages.”
That framing is a decision about whom the system serves. Shannon says the system “must be designed to operate for each possible selection, not just the one which will actually be chosen since this is unknown at the time of design.” An engineer building a telephone line cannot know in advance which sentence will be spoken on it, so the line has to carry any of them. What it can use is statistics: some messages are more likely than others, and the paper extends the earlier theory to cover “the savings possible due to the statistical structure of the original message and due to the nature of the final destination of the information.”
That last phrase deserves a second look. The destination is already present in 1948, as one source of savings among others. Shannon names it without making meaning the object of the theory. Leaving meaning out made the problem solvable, and in exchange it left open a question: when a record has to lose something, who decides what?



