This is Part 2 of Mathematical Thinking for AI, a series on the twelve ideas underneath modern machine learning, and what each one costs.
The idea
Linearity. In a linear world, the whole is exactly the sum of its parts, no more, no less, and nothing appears in the combining.
The picture
Linearity is a world without surprises.
Do two things together and the result is exactly what you would have gotten doing them separately, then adding. Scale the input by ten and the output scales by ten, no more, no less. Nothing interacts. Nothing emerges. Add the causes and you have added the effects, forever, in every direction, at every scale.
The world is not like this. Stretch a spring twice as far and, soon enough, it gives rather than pulling twice as hard. Double a drug dose and you do not double the effect. Put two people in a room and what happens is not the sum of what each would do alone.
We know all this. And we insist on pretending otherwise, everywhere, all the time, because a world without surprises is the only world we can compute in.
And “compute” here means something specific and humble. In a linear world, you can understand a system by interrogating it one simple input at a time. Feed it pure cases, record what comes back, and you are done: any complicated input is a mixture of your pure cases, so its output is the same mixture of your recorded answers. Divide, probe, recombine. Whole fields of engineering that speak of “response” lean on this, and so does every unit test you have ever written against a system you hoped was modular. The hope that testing parts tells you about wholes is the hope of linearity, wearing work clothes.
That is the claim from the top, now with the picture behind it: in a linear world, the whole is exactly the sum of its parts, no more, no less, and nothing appears in the combining. The whole gift, and the whole problem.
The problem it answers
The problem is that description explodes.
A box sits on the bench with a thousand switches along its front and a thousand gauges along its top, each gauge showing a single number. Set the switches any way you like and all thousand gauges settle on new values: a pattern of a thousand on and off switches goes in, a list of a thousand numbers comes out, and nothing is known about the wiring between.
To say what this box does, in full, you would record all thousand readings for every pattern the switches can take, and the patterns number two to the thousandth power, more rows than there are atoms in the observable universe to store them on. For almost every such box no shortcut exists. The table is its own shortest description. Nobody is hiding the function from you: there is nothing smaller than everything that could be handed over.
Now assume the box is linear. In this box’s own terms the assumption says one thing: whatever two switches do together is exactly what each does alone, added up. No switch changes what another is worth. No pair conspires to move a gauge that neither moves by itself.
The number collapses. What the box does is now fixed by one million numbers: for each of the thousand switches, how much it contributes to each of the thousand gauges. A thousand times a thousand. Not approximately fixed. Completely. Every question you could ever ask about the box is answered by that modest grid of numbers.
And the million are not an arbitrary million. They are what the box does under a thousand experiments. Flip the first switch alone and write down the thousand readings. Then the second, and so on to the thousandth. A thousand experiments, each producing a column of a thousand numbers, and the columns side by side are the million. From them you know what the box will read for all two-to-the-thousandth patterns you will never try.
Now replace the switches with knobs that turn to any position. The settings stop being countable, and the same million numbers still cover every one. On and off switches needed only the adding half of the assumption. Knobs need the other half: a knob turned three tenths of the way contributes three tenths of what that switch contributed at full, and contributions still add. So any setting whatsoever is a weighted sum of the thousand you measured, and its readings are that same weighted sum of the thousand columns you wrote down.
That box is not a thought experiment. A thousand switches, a thousand gauges, a million numbers between them: that is one layer of a neural network, and the million numbers are its weights. Linearity converts exhaustive enumeration into finite measurement. You do not need to visit the space. You need to visit a skeleton of it, and the assumption fills in everything between the bones.
From more-entries-than-atoms to a thousand probes. That is what the assumption buys, and it is why the assumption gets made whether or not it is true. Linearity was never adopted because the world is linear. It was adopted because without it, description cannot even begin.
In December 1807, Joseph Fourier read a memoir on the propagation of heat to the Institut de France in Paris. His central move was an act of decomposition: any initial distribution of temperature, however jagged, is a sum of sine waves. The complicated thing is a weighted pile of simple things. Understand the parts, add them up, and you have understood the whole. That move has a name, superposition, and Fourier applied it with a straight face far beyond where it had any license yet to go.
The committee was four men: Lagrange, Laplace, Monge, Lacroix. Three of them were willing to see the memoir published. Lagrange was not, and the memoir stayed unpublished for more than a century and a half.
The cheap version of this story is genius suppressed by conservative authority, and the cheap version is wrong. Lagrange had studied trigonometric series himself, decades earlier, in work submitted to the Berlin academy. He had handled the coefficients. He knew where the convergence questions lived. He was not failing to understand Fourier. He understood the road better than Fourier did, and he knew Fourier could not prove the road was safe. The committee’s demand for a convergence argument was mathematically legitimate, and Fourier could not meet it.
What Fourier was betting on was something else entirely: a representation useful enough deserves to be used before it is justified. He won the bet. The heat equation gave way, sine decomposition became the common tongue of the physical sciences, and the rigorous conditions arrived from later hands, after the tool had already reshaped the field. The work appeared at last in 1822, substantially rewritten, as his treatise on heat, fifteen years after the room in Paris said no.
What survives the biography is the wager. Superposition worked before anyone could prove it should. Deep learning stands today in a position Fourier would recognize. Partial theories exist and multiply, but nobody can prove, in the sense Lagrange would have accepted, that the whole arrangement should work. The proofs trail the practice by years, where they exist at all, and the practice pays anyway. If that offends you, Lagrange is your man, and his objection was never answered. It was outrun.
The cost
So linearity buys computability at a discount that cannot be refused. What does it charge?
Everything interesting is nonlinear. True, and useless. Everyone knows it, which means it is not a cost anyone is failing to price. It is a complaint.
The real charge sits under that one. A linear system cannot make anything new out of the meeting of its inputs.
It can transform, and transformation can look creative: rotate a representation and unfamiliar coordinates appear, project it and structure you had not noticed stands out. But all of that is the old material rearranged, and the rearrangement was fully determined the moment the map was written down. What a linear system cannot do is let its inputs matter to each other.
Every output of a linear map is a weighted mixture of its inputs, each input contributing on its own, none of them ever meeting. No output can depend on two inputs jointly, on their agreement or their conflict, on one being large while the other is small. The map can amplify, attenuate, rotate, and blend. It cannot let ingredients react. In a linear world there is mixing but no chemistry.
The wall is easy to walk into. Build, out of weighted sums alone, a detector that fires when exactly one of two switches is on and stays quiet when both are on or both are off. It cannot be done. Any weights you pick that reward each switch alone, above the all-off baseline, will reward the pair even more. The function is easy to state, easy to want, and flatly outside the vocabulary, because it is a function of the switches’ relationship, and relationships are the one thing weighted sums cannot see. Geometrically: draw the four corner cases on paper, and no single straight line separates the ones that should fire from the ones that should not.
And this has a consequence you can verify on the back of an envelope. Take a neural network one hundred layers deep and delete every activation function, leaving only the linear maps. The whole tower collapses into a single layer. Not approximately, not in the limit: the composition of linear maps is itself a linear map, and a single layer can replace the entire stack, exactly. One hundred layers of depth, and depth bought nothing.
So a neural network is a stranger object than it looks. The parameters, the bulk, the trillions, live in the linear maps. But the thin films of nonlinearity between them, the activations, the softmax, the normalizations, cheap and almost dismissively simple, are the only places in the entire system where chemistry is allowed to happen. A linear map cannot let one input change what another is worth, since every weight is fixed before anything arrives. A threshold can: whether a signal passes now depends on the value it carries, so what one input contributes can turn on what another already did. That is mixing versus reacting, and it happens only in these thin joints. Remove them and depth is an illusion.
The system spends essentially all of its parameters on the part that cannot create anything, and creates everything in the part that has almost none. The linear maps are where learning is stored; the nonlinear films are why there was anything to learn. Neither is the interesting half on its own. The whole enterprise of deep learning is a negotiation, repeated at every layer, billions of times per second: linear so we can compute it, nonlinear so it was worth computing. Every architecture you have ever admired is a particular set of terms for that negotiation, and most architectural history is the story of those terms being renegotiated.
And linearity does not fail the way other assumptions fail.
Linearity fails silently. Fit a linear model to a system whose parts genuinely interact, and the model does not break, does not warn, does not refuse. It smears the interactions into its coefficients and into what it calls noise, and then it hands you an answer: clean, interpretable, plausible, wrong.
Most assumptions fail silently. A misspecified model rarely announces itself. What makes linearity’s silence dangerous is its eloquence. The failed model still explains itself beautifully: every coefficient means something, every effect has a size, the story is coherent end to end. The one thing the story cannot contain is the very thing that mattered, because the vocabulary has no word for two inputs meeting. It will not tell you it failed. It cannot know.
Where it lives today
First: nearly all of the parameters you have ever trained live inside linear maps. Walk through a transformer and check which components carry weights. The query, key, and value projections: linear. The output projection: linear. The two big matrices of the feed-forward block: linear. The embedding in, the unembedding out: linear. And the nonlinear organs, the activations and the softmax, carry no parameters at all, save the small scale factors in the normalization layers. This is not a research finding. It follows from the architecture’s definition, and anyone can check it against the code in an afternoon. By weight, a transformer is overwhelmingly a linear object, with nonlinearity applied in thin, parameter-free films at the joints.
Second: the economics of the industry are downstream of the same fact. A matrix multiply is one small operation, multiply two numbers and add the result to a running total, repeated millions of times. No repetition waits on any other, so millions can run at once. And the chip knows in advance which numbers it will need, because they are chosen by position, say the third row against the fourth column, rather than by anything it has to compute first. It can fetch the next batch while the current one is still being multiplied, and nothing ever stalls waiting to be told where to look.
That shape is what silicon can be built for. Hardware does not accelerate difficulty. It accelerates regularity, and linearity is where the regularity is. Tensor cores are built for precisely this workload. And because the workload is nearly all one operation, the industry can price everything in a single number: the FLOP, one arithmetic operation, counted by the trillion. The count means something only because every operation costs about what every other one costs, so many FLOPs converts directly into so much time, so much electricity, so much money. Had the workload been a mixture of unlike operations, the same count would buy different amounts of work depending on the mix, and it would measure nothing at all. Instead the conversion rate holds from one chip to a hundred thousand, which is what lets a single unit stretch across five orders of magnitude. A scaling curve needs an axis whose units mean the same thing at both ends, and this is where that axis comes from. An entire capital cycle, the largest infrastructure buildout in the history of computing, rests on the fact that intelligence, as currently practiced, spends almost all of its arithmetic inside the one mathematical structure we know how to accelerate. The chips are shaped like the idea.
Third: LoRA, and its bill.
LoRA takes the low-rank route to cheap fine-tuning. Rank is a linearity idea, and the box makes it concrete. If the fortieth column is just the third and the ninth in some mixture, scaled and added, it tells you nothing those two did not. Rank counts the columns that are genuinely new: a grid of a million numbers has low rank when a handful account for all the rest. The premise, and the original paper is candid that it is a hypothesis, is that when a large model adapts, the change in its weights has that character: a big matrix of low rank, the product of two thin ones. Train only the thin pair and the parameter count falls by orders of magnitude.
It works, which is evidence about how these models change: the linear structure the hypothesis bets on is there, often enough to build an ecosystem on.
In 2024, someone itemized the bill. A study compared the weight matrices LoRA produces against those from full fine-tuning, and found a structural difference. Full fine-tuning spreads its change across many directions, each nudged a little. LoRA cannot: the assumption allows only a handful, so the same change is forced through them, and each moves a long way. What comes out is a few powerful new directions that dominate the update and sit almost at right angles to everything already there. The authors called them intruder dimensions, then intervened on the finished model to see what they do. Scale those directions down and the model gets back most of the general-text performance it lost when it learned the new task. It does not end up better than the model it started from; it recovers some ground. Meanwhile the narrow task it was tuned for, contract summaries or ticket classifications, loses very little. That is causal, not correlational: the forgetting was not spread through the model but sitting in those few directions.
The linear assumption, that the update is low rank, bought a discount of orders of magnitude in trainable parameters, and it was real. The task benchmarks showed no charge, and were not lying: on the new task there was none. The charge fell on the model’s memory of its old life, everything it knew before it learned the new task, and it fell in a form no training run reports. To find it you break the weight matrix into its independent directions and read how far the matrix stretches along each. That list of strengths is the spectrum. It is not exotic; researchers compute one whenever they need to compress or analyze a matrix. But computing it before and after a run and comparing them was not part of anyone’s workflow, which is why the charge went unnoticed while the method spread.
And it can be paid. The remedy is the intervention the researchers ran, and it is cheap. What it costs is the thought of looking, which depends on how you hold the idea. Most people hold LoRA as a fact about parameter counts: it trains a thousandth as many numbers. True, and it suggests nothing to check, because parameter counts have no directions in them. The other version is a claim about shape: the change this model needs can be built from a few directions. Hold it that way and the diagnostic is immediate: if the update is supposed to live in a few directions, look at the directions and see which it used.
The linear assumption left a fingerprint, and it formed in a place the task benchmarks were never pointed at. The charge was not on the benchmark the method was sold on, but on a capability the assumption had quietly mortgaged, in the model’s memory of everything it knew before.
One more home, far from machine learning. A doctor treating three conditions at once is doing addition: one drug for the blood pressure, one for the cholesterol, one for the sleep. The arithmetic works and medicine could not function without it. Then two interact and produce an effect neither produces alone. Wherever a system is understood as a sum of separately understood parts, the failures wait in the interactions, invisible to anyone studying the parts.
Take it with you
Two probes, testing different things. The first asks what a design can express. The second asks what an assumption costs.
Expressive power is decided by where inputs are allowed to act on each other, not by parameter count and not by depth. One honest exception: widening the maps buys capacity, but that claim is scale, and belongs beside other ways of spending the compute.
When a paper announces an architectural breakthrough, ask where the new interaction is. Mark every run of consecutive linear steps in the diagram: two projections back to back, a reshape between two matrix multiplies, a transformation ahead of an existing one. Each run collapses to a single map however deep it looks. If the whole change lives in such runs, the paper has a story about compute, memory, or gradient flow.
For comparison, a real interaction site. In a linear layer every coefficient is a constant fixed when training ends: the third input is multiplied by its number, the seventh by its own, and nothing the third does changes the number applied to the seventh.
Attention breaks that. Reading a sentence, the word “it” draws on whichever earlier word it stands for, and which word that is depends on both. So attention compares “it” against every earlier word and uses the strength of each match as that word’s multiplier. The coefficient is no longer a constant. It is computed from the pair, which a fixed matrix cannot do.
Which is why the two must be kept apart: a faster or cheaper design is a claim about cost, and only a new meeting place is a claim about capability. A line reporting a faster architecture with stronger reasoning is two claims in one breath, and that is how abstracts get written.
A discount is never bought with efficiency. It is bought with an assumption about the shape of the world, and the assumption is what will eventually be wrong. So when anyone offers you almost the same quality at a fraction of the cost, ask which assumption the discount is coming from. Somewhere a structural bet has been signed, low rank, locality, separability, some cousin of linearity, and the bet has a blind spot by construction, because that is what an assumption is. Find the assumption and you have found the address where the bill will eventually be delivered. LoRA’s arrived in the spectrum of the weights. Yours will arrive somewhere specific too, and the vendor has not looked.
Linearity is the first idea of this course for a reason. It is the purest case of the trade every mathematical idea makes: a world without surprises, exchanged for the power to compute, with the surprises smuggled back in through thin films at the joints, and a fingerprint left everywhere the exchange was signed.
Mathematical Thinking for AI. State the idea. Read the trade. Find it somewhere else. Twelve mathematical ideas that decide what a machine can know.