This is Mathematical Thinking for AI, a course in the twelve ideas underneath modern machine learning, and what each one costs. No formulas.
You can derive backpropagation. You can write down the Adam update from memory. You can explain, without hesitating, why the normalization layer goes where it goes.
So here is a question. The gradient, as an idea: what did it make you give up?
Most people cannot answer. Not for lack of intelligence: nobody taught it from this angle. We teach the method, the derivation, the implementation, then send people into a field where every serious decision rests on something never taught.
Formulas can be looked up. Ideas have to be worn in. Your judgment lives in the layer that cannot be outsourced.
What you are missing is not mathematics
Three layers, and not the same kind of thing.
The third layer is routinely misread, and the misreading is always the same one: mistaking it for a fact you have not yet learned. You are almost certainly not missing facts. An idea is a truth you have pushed so far that it stopped being knowledge and became a reflex: a question you now ask by default.
A fact you can recite but have never reasoned from is, in every observable way, the same as a fact you do not have. It changes nothing you ask, nothing you check, nothing you notice, and nothing you build. It sits in your memory and is absent from your work.
A field can do the same thing at scale: know an idea and still not use it, until someone puts it where it belongs. That is the pattern behind most of the field’s real breakthroughs:
The important breakthroughs in deep learning rarely come from new mathematics. They come from taking an old idea and putting it in the right place.
That does not mean mathematical innovation is unimportant, or that new methods are old methods repackaged. Genuine mathematical inventions arrive, and reshape the field when they do. What it means is more precise, and more useful: the layer you actually reason from when you make a judgment, the one that tells you why a method should work and when it will stop working, almost never comes from the newest mathematics. It comes from a small number of very old ideas that can be understood without symbols at all.
Those ideas are what this course teaches. Twelve of them.
What mathematical thinking is
Mathematical thinking, as this course uses the phrase, is not a feeling or an aesthetic. It is three things you do, and they form a chain rather than a checklist. What you decide the technique is claiming determines which questions become available; the questions expose what was traded away; the shape of that trade is what lets you recognize the same idea somewhere it has no business being.
One: state the idea underneath the technique.
Ask what kind of object gradient descent is, and the useful answer is not “an optimizer.” That is the shelf it is stored on, not what it is. Underneath, it is a position on what can be known: global information about a landscape is unaffordable, so a step taken on local information is not a compromise but the only move you can afford.
That sentence is the claim. An algorithm cannot be false; it can only run. A claim can be false, and that is exactly what makes it valuable, because only things that can be false can be questioned: what does this cost, where else does this hold. Those are the next two actions, and neither has anything to grip until the claim exists.
Three things are doing work here. The technique has a name, gradient descent, a label for an algorithm that runs and nothing more. The idea underneath it has a name too, the gradient, and that name is just as much a label: a way to point at the idea, not to state it. What carries the actual content is the claim, not either name, because a sentence can be wrong and a name cannot.
How do you know you are holding the claim and not just the name? It fits in one sentence, and it could turn out to be false. Linearity, the metric, the gradient are names, good for pointing at an idea, incapable of stating it; if the name is all you can produce, you are holding a technique wearing an idea’s clothing. Most people fail this on the first try, with a method they have used for years.
Two: read the trade.
Every mathematical idea answers a question in advance, on your behalf, before you have thought to ask it. That is what makes it powerful and it is what it charges for. Answering in advance is the whole service, and it is the whole charge: a question answered in advance is a question you no longer ask. So the test is subtraction. If you cannot say what an idea took away, you have not found the idea.
Take the metric, two parts from now. Represent meaning as position, and similarity becomes distance, and distance is computable. Every embedding model and vector database follows from that agreement.
The charge arrives at the collapse. Suppose you are matching resumes to a job opening. One candidate sits near your best hires by skill overlap and near your costliest ones by tenure pattern, and a rich representation holds both truths at once. But ranking asks for the nearest match, and nearness folds into a single number, and a single number imposes a single ranking. The fold is the decision. In choosing how many resemblances compress into one score, you decided, before any query arrived, which kind of likeness survives the compression and which silently vanishes inside it.
That bill did not arrive in the representation, rich enough to hold both. It arrived in the single motion that turned it into one number, which nobody treats as a decision. Costs rarely land where the idea was adopted. They land wherever the question you were prevented from asking gets asked anyway: a strong candidate scored low because tenure risk and skill match had been folded into one number, discovered only when someone asks why the team keeps losing people who looked identical to this hire on paper. Nothing warned you, and nothing was going to.
Three: find it somewhere else.
An idea that appears in only one place is not an idea, it is an explanation. Duality shows up in constrained optimization, in the actor and the critic, in the structure of a market. That is neither coincidence nor analogy. Duality is a fact about any problem where something scarce is allocated under constraint, so it was never a fact about optimization or markets at all. It constrains the space of solutions, and the fields sit downstream of it.
Recognizing that structure is what lets you find it somewhere else, and finding it there is what turns a fact into a tool. It tells you what must be true of a new case before you study it, including what it will cost. See it three times, and the fourth time you recognize it before anyone names it, and know what to check.
State it. Read the trade. Find it elsewhere. Three actions, twelve ideas. Each decides something specific about what a machine can represent, optimize, or never reach at all.
Here is the first, and it is the one you are most confident you already understand. Every part of this course is built the way this one is: a picture and the problem it answers, which state the claim; the cost and where it lives today, which read the trade; and a closing section that puts the idea to work somewhere it has no business being.
Demonstration: the gradient
The picture
You are on a hillside, in fog. You cannot see the valley, only the slope under your feet.
The idea of the gradient: use only the information in this one small patch of ground to decide where to step next. That is the whole thing. Everything else is bookkeeping.
The fog is not a failing of the hiker. It is made of the size of the space, the cost of looking, and how little one query returns. No instrument fixes it and no budget clears it. Where this idea is actually used, the valley could not be seen from where you stand, by anyone, with any equipment.
Accept that, and the gradient stops looking like a compromise and starts looking like the only proposal on the table. Which is how it arrived.
That is the claim of the gradient, said again with your own feet in the fog: global information about this landscape is unaffordable, so a step on local information is not a compromise, it is the only move you can afford.
The problem it answers
In 1847, Augustin-Louis Cauchy submitted a short note to the Académie des Sciences in Paris, titled, in translation, A general method for the resolution of systems of simultaneous equations.
He was computing the orbit of a heavenly body, not by solving the differential equations, but by attacking the algebraic equations of the motion directly, six orbital elements as the unknowns.
The standard procedure was successive elimination: grind the system down to one equation, then solve it. Cauchy’s objection was blunt. In a great many cases, he wrote, the elimination cannot be carried out at all, and when it can, the resulting equation is generally very complicated, even when the ones you started with were simple.
So he proposed something else. Not a route to the closed-form answer, but a way to keep improving a guess, using only local information, as long as you care to continue. He even suggests finishing with a faster method once the guess is good, which reveals what he abandoned. Not precision. The demand for a direct solution.
Gradient descent was born as a decision to stop solving and start descending. It arrived nearly a century before the first computer, forced into existence by problems, like fitting an orbit, where the classical route could not be walked.
Our situation is far more extreme than his.
In a few dimensions, giving up global information is a convenience. In our spaces it is not a choice. The landscape exists as a mathematical object, every point of it, and that is no help: no feasible procedure inspects more than a vanishing sliver. The global map is not hidden. It is unaffordable, at any budget you will ever have.
The gradient is not one option among several. It is the only one whose price does not explode.
Suppose you did without it: guess a direction and nudge along it a small amount, then ask only one question: did it help or hurt? Keep the change if it helped, back off if it did not, then guess again from wherever you landed. A whole family of methods works exactly this way, and in two dimensions it works fine: there are a handful of directions worth trying, and testing one costs almost nothing.
A finer version of the same idea exists, and at this scale it works just as well. Instead of asking only whether the nudge helped or hurt, ask how much it moved the needle: nudge a small amount in some randomly chosen direction, measure the exact change, return to where you started, nudge the same amount the opposite way, and measure again. The difference between those two readings gives the slope along whichever direction you chose, not just its sign. With two or three dimensions this costs almost nothing too: a handful of directions, two readings each.
However, a model with a billion parameters has a billion dimensions, and a direction is a complete specification of which way to move every one of them and by how much. Guessing a direction means guessing a billion coordinated numbers at once.
Now the geometry of such spaces, and it is nothing like the geometry of the three-dimensional room you are sitting in. Draw two directions at random. In three dimensions, their alignment, the same cosine similarity used throughout this course, could fall anywhere between fully aligned and fully opposed, with no value more likely than any other. In 768 dimensions, small by current standards, the typical alignment between two random directions, measured the same way, is around 0.036, and the number keeps shrinking as the dimension grows. They are, almost always, very nearly perpendicular, not because perpendicular was ever favored but because at high dimension almost nothing else is left possible. Point in a random direction and it tells you almost nothing about the one you want. Perpendicular is what zero information looks like: project one direction onto a direction exactly perpendicular to it and nothing comes through at all. It is the way walking sideways across a hillside reveals nothing about how steeply it drops straight ahead. Nearly perpendicular means nearly nothing comes through. The two barely speak.
That defeats the coarse version of guessing, which only ever learns whether a nudge helped or hurt, kept or discarded on faith. It defeats the finer version too, for the same reason, extended by one step. The true gradient is a single fixed direction, not a random one. But the argument never required both directions to be random. Strip the length off any direction and what is left is a point on the sphere of all possible directions, the gradient’s included, and that sphere looks identical from every point on it. So a direction drawn at random is nearly perpendicular to any fixed one you measure it against. A nudge along that random direction therefore measures almost none of the gradient’s actual slope.
Recovering the whole gradient this way is a system-of-equations problem: a billion unknowns generally need a billion independent equations to pin them down, and each round of nudging supplies exactly one equation. Absent special structure in the gradient, which nothing here entitles you to assume, there is no shortcut: you need on the order of one such round per dimension. At a billion parameters that is not an inconvenience. It is a wall.
Which means the fact of which way is downhill is, in high dimensions, so rare that guessing for it is ruinous. There is another way to get it, one that never has to guess at all. Reverse-mode differentiation hands you that fact for every parameter at once, for roughly the cost of running the model a handful of times. It works backward from the error, retracing the exact chain of calculations that produced it and assigning each parameter its share of the blame along the way, all in a single backward sweep rather than one round per parameter. That is the bargain the industry was built on: one pass forward to see what the model currently answers, one pass backward, and out comes the individual verdict on every one of a billion dimensions, which way each should move. This collection of verdicts, one per dimension, is what the whole demonstration has been circling: the gradient itself, the fact of which way is downhill, handed to you whole instead of guessed at one direction at a time. Each dimension is then nudged a small step in the direction its own verdict recommends, the error drops a little, and the whole cycle, forward pass, backward pass, a billion small nudges, repeats from wherever the parameters now sit. Other routes exist, and without special structure they pay more for less, by a factor that grows with every parameter. We have spent a decade adding parameters.
The cost
So you took the deal. What did it cost?
You went blind to everywhere except here. That sounds right. It is not.
Suppose the landscape is a single smooth bowl, one valley, no ledges, no flats. Then local information is enough to guarantee arrival: the slope under your feet may not point straight at the answer, but following it, at a sane pace, gets you there from anywhere, blindfolded. Not seeing the far landscape never puts the valley floor out of reach. So being unable to see the rest of the landscape is not what the fog charges you for.
The real charge falls even inside that one forgiving valley, and it has nothing to do with what lies far away. The fog was never the thing charging you. The fog explains why you must walk by feel; it is the gradient, the particular instrument you reached for, that decides how much you get to feel. The gradient is a first-order object. It reads the slope in your immediate vicinity and reports one thing: the steepest direction, and how steep. That is all it keeps. How the ground curves, whether it falls away evenly in every direction or drops hard in one and barely at all in another: all of that is structure in the terrain right where you stand. The gradient projects every bit of it away to hand you one arrow.
What the gradient charges you is this. To buy a direction you can compute at a trillion parameters, you agree to see the terrain at the coarsest possible resolution: one direction, and nothing of the shape that produced it.
This is not blindness to far-off places. It is blindness to the place you are standing. The curvature under your feet is real, local, knowable in principle, and discarded by construction. The whole shape of the bowl falls outside what one arrow can carry.
Two places on the landscape can tilt at the same angle and be nothing alike. Land in a wide shallow basin and the parameters can be nudged in any direction with almost no effect on the answer; land in a narrow crevice and a small nudge wrecks it. Same tilt, opposite fates, and the model has no way to tell you which one it is in. What separates the two is not how steeply the ground slopes but how fast that slope changes as you move, and the gradient reports only the slope.
The wide landing tends to hold up better on data the model has never seen, which is close to the whole reason anyone trains a model. So the property people care most about is a property of the shape, and the shape is what the gradient discards to buy its price. How stable a solution is, how wide its basin, how far you can move before it breaks: none of it is in the arrow.
The consequence people notice comes later. Run the identical procedure in one smooth bowl and in a landscape of many basins and it cannot tell them apart: the loss goes down in both, and there is no term anywhere inside it that reports which world you are in. But that is a symptom, not the disease. The gradient never had the resolution to report the shape of a world at all, only the direction down from where you stand.
This is not a defect a cleverer optimizer removes. It is the definition of the idea: a first derivative keeps one direction by discarding the rest. Cauchy signed this contract in 1847, on your behalf, and you have lived under its terms since. He solved for six orbital elements and could hold the whole surface in his head. We solve for a trillion, with an arrow.
Where it lives today
Where you start shapes where you can end. Change the seed, land somewhere else, get a different model. What was in the basins you did not enter, this run will never tell you. People say this constantly: we tried that architecture and it did not work. Sometimes that means the architecture cannot work. Sometimes the capacity was there and this optimization missed it. Sometimes the data, budget, or evaluation was the problem. A failed run is evidence about a training configuration, not an impossibility proof about an architecture, and the gradient cannot say which.
Models are strong near their training distribution and fall apart away from it, and the gradient looks like the reason. It is not. That failure is not the gradient’s bill. It is the bill for learning from a finite sample. Change the algorithm to anything you like, gradient-free, exact, not yet invented, and the conclusion does not move: where the data never went, there is nothing to have learned from. What the model does out there comes from its inductive bias: the assumptions built into the model that decide what it guesses where the data is silent. A convolutional network guesses a shifted pattern is the same pattern; the data never said so. That is a different idea with a different bill. The gradient merely happened to be in the room. Attaching a cost to the wrong idea feels exactly like understanding, and is worth less than nothing, because it is confidence aimed at a false cause.
What was discarded gets approximated back, piece by piece, but never past what scalability still allows. Momentum, Adam, learning-rate schedules: each is a cheap partial estimate of how the ground curves, costing an extra number or two per parameter, the same order as the gradient itself. They recover a trace of it from the sequence of gradients already collected, reading the curve from how the slope has been changing step to step. It is a little of what was thrown away, bought back cheaply enough to keep the original deal standing.
Only the exact method returns it in full, and full has a price: a matrix sized by the parameter count squared, which at a billion parameters is a quintillion numbers, and using it means solving a system of that size at every step. That is the same class of computation Cauchy walked away from in 1847, arrived at from the opposite direction. The gradient was invented to escape exact solving; buying back what the gradient discards puts you in front of exact solving again.
Take it with you
Next time a gradient-trained system impresses you, ask this, because a first-order method has no way to answer it for you:
Which properties of this model does the training objective actually demand, and which are just accidents of the one basin this particular run happened to land in?
The gradient cannot tell the difference. It reports the way down from wherever you are standing, never the shape of the terrain that put you there, so it has no way to certify that what you are looking at is necessary rather than incidental. The first kind of property survives a change of seed. The second belongs to one run and gets reported as a property of the method. The answer changes from system to system, and tells you more than the benchmark table will.
That was not a lecture on gradient descent.
You stated the idea: a step on local information, in a world whose global map is forever unaffordable. You read the trade: scalability, bought by seeing the terrain at the coarsest resolution, one direction and nothing of the shape that produced it, a bill that comes due as a seed that quietly decides a model’s destiny and as the whole folk craft of training schedules, buying back in pieces what one arrow threw away. And you found where the idea does not belong: correcting the reflex to blame it for a failure, generalizing badly to new data, that was never its bill to pay, which matters just as much as finding where an idea does.
That is all three actions, on one idea. Now, eleven more times.
The curriculum
Four modules. Each asks a single question.
Representation: how do you turn a world into numbers? Linearity. Metric. Invariance. Every choice here precedes training, invisible in the loss curve, permanent in the result.
Uncertainty: how do you act without knowing? Confidence. Surprise. The tail. Why the average is the most reliable liar in the field, and why what breaks your system was, in expectation, never going to happen.
Optimization: how do you get better? The gradient. The landscape. Duality. You just saw the first hour of this one.
Limits: what is impossible? The fixed point. Inductive bias. Self-reference. The most expensive mistakes in this industry come from people who never seriously considered that some things cannot be done, and so cannot tell a hard problem from a closed one.
Each module closes with a Studio: one real system taken apart with the three ideas just learned. A self-driving car's camera and the leaderboard that ranks it. A scaling law and its tail. A post-training pipeline. A model that claims to reason.
Who this is for, and who it is not for
This is not for you if you want to learn mathematics. No proofs, no exercises, almost no equations at all. Nor a history of mathematics: history appears only when it explains why an idea was forced into existence, then it leaves. Nor meditations on mathematical beauty, though it does turn up: an idea that explains three unrelated things at once is a beautiful object, and you will meet several. What these ideas let you see, and what they charge for the seeing, is the subject.
This is for you if you make technical judgments for a living, and have noticed they rest on intuition, analogy, and things other people said, and not one thing you could state clearly and defend.
Uncomfortable, normal, fixable, and never fixed by reading another paper.
What you will have
At the end of this you will not compute anything faster. No technique here executes better than the one you already use. What changes is which problem you decide to solve, and what you expect to break.
What you will have is three actions, in a particular order.
You will look at a new method and see, under the architecture diagram, the idea it runs on, the trade it made, and where to look for the bill. And later you will meet the same idea in a robot, in a market, in an argument that has nothing to do with machines, and recognize it, and know what to check.
The third action changes what you are able to think about. The other two are how you get there.
Mathematical Thinking for AI. State the idea. Read the trade. Find it somewhere else. Twelve mathematical ideas that decide what a machine can know.
Hugo, I am not a math guy, but to me this is an elegant explanation of the gradient as a way to move through a landscape whose global shape is too costly to see. My understanding is that repeated local steps—and attention to how their results vary—can recover some information about the surrounding curvature. But they cannot guarantee an accurate picture of the entire second-order shape. Repeated gradient-guided experiments can recover some information about local curvature, but they remain trajectory-dependent. They may enlarge the area understood without revealing the full landscape or proving that another basin would not have produced a better result. The process may only map a larger local region, or even become a feedback loop that repeatedly confirms the path it has already taken.
What if we act as flora, not fauna?
Then there is no direction, destination, goal. It is simply taking energy and resources available and using them to expand our ability to generate and accumulate energy and resources, while investing and incorporating them back into our environment in ways that sustain it as well.
If we don't act as reasonably healthy components of that larger ecosystem, then we are parasites and will eventually elicit an immune response.
While this might simply seem esthetic, it is how people function in mass. As nations, groups, even as corporations in a larger economic ecosystem. As cells within these processes, we are at most feelers, or structural components.
So then is our role simply to reenforce that group effort, or should we look to the larger context, even if it should contradict our immediate relations. When the feedback loops turn negative, trying to counter them often ends up as just spinning the wheels deeper into the mud.
Irrespective of all those gradients, the one signal we all share on the surface of this little orb, is that center of gravity that has all of us standing perpendicular to the surface. While it might seem beyond our various strategic models, it is many ways, the elephant in the room.
Where is life and humanity going.