This is Part 3 of Mathematical Thinking for AI, a series on the twelve ideas underneath modern machine learning, and what each one costs.
The idea
Invariance. It is what a system refuses to learn, and the only question is whether the refusal was chosen or inherited.
The picture
Every system, before it learns a single thing, has to answer one question: which different inputs count as the same input?
A word in capitals and the same word in lowercase. A photograph shifted three pixels to the left. A sentence set in a different font. A transaction recorded in 1990 dollars or in 2020 dollars. Each of these differences either counts or does not count, and the system has to pick a side, because it cannot carry every distinguishable difference forward. A representation that preserved everything would be the world itself, and useless for the same reason a map at full scale is useless.
An invariance is that decision, made and enforced: this entire family of differences is noise.
And the decision is rarely experienced as a decision. Somebody added a lowercasing step to a preprocessing script in 2019 because it improved a benchmark score, and from that afternoon onward the system could not tell Apple from apple, which is fine for sentiment and fatal for entity resolution. Nobody convened a meeting. The line went into a file, the file went into a pipeline, and a permanent constraint on what the system could ever distinguish was established by a person who was thinking about something else.
Which is the claim from the top, now with the picture behind it: an invariance is what a system refuses to learn, and the only question is whether the refusal was chosen or inherited.
