Mathematical Thinking for AI, Lesson 3: Invariance
Before a system learns anything, it decides which differences are noise. That decision cannot come from the data, and it sets a ceiling nothing downstream can raise.
The picture
Every system, before it learns a single thing, has to answer one question: which different inputs count as the same input?
A word in capitals and the same word in lowercase. A photograph shifted three pixels to the left. A sentence set in a different font. A transaction recorded in 1990 dollars or in 2020 dollars. Each of these differences either counts or does not count, and the system has to pick a side, because it cannot carry every distinguishable difference forward. A representation that preserved everything would be the world itself, and useless for the same reason a map at full scale is useless.
An invariance is that decision, made and enforced: this entire family of differences is noise.
And the decision is rarely experienced as a decision. Somebody added a lowercasing step to a preprocessing script in 2019 because it improved a benchmark score, and from that afternoon onward the system could not tell Apple from apple, which is fine for sentiment and fatal for entity resolution. Nobody convened a meeting. The line went into a file, the file went into a pipeline, and a permanent constraint on what the system could ever distinguish was established by a person who was thinking about something else.
In one sentence: an invariance is what a system refuses to learn, and refusing is not optional. The only question is whether the refusal was chosen or inherited.
