Reinforcement learning was not invented in 1988. It was not invented in the 1950s either, and it did not begin with a psychologist watching a cat escape from a puzzle box. The loop that makes it work, sense the world, compare what you got against what you expected, act on the difference, was already running in a single cell before there were animals, before there were neurons, before there was anything on this planet that could be said to have a nervous system at all.
That is the claim of this chapter, and it is not a metaphor. The bacterium Escherichia coli runs the loop with proteins. It carries a memory. It forms an expectation. It computes the difference between what it expected and what it found, and that difference, not the food itself, is what drives its behavior. Strip a modern learning algorithm down to its load-bearing parts and you find the same skeleton, built out of different material.
So the question that organizes this chapter is not whether a bacterium is intelligent. It is a more useful question: what is the minimum a system needs in order to predict its world, and what does the cheapest possible answer look like?
It looks like this.
The problem: a cell cannot see space
Start with the difficulty, because the difficulty is what makes the solution ingenious rather than obvious.
An E. coli cell is a rod about two micrometers long. Somewhere in the water around it there is sugar, denser in one direction, thinner in the other. The obvious way to find the sugar is to do what you would do: sense the gradient, notice which side is richer, and swim that way. Compare here to there, then steer.
The cell cannot do this. There is a real gradient across the length of its body, but it is vanishingly small, and the cell lives in a world where everything is being kicked constantly by thermal noise. Molecules arrive at its receptors at random. The difference in arrival rates between the front of the cell and the back is buried under that randomness. To dig the signal out, the cell would have to sit still and average its measurements for a long time, and by the time it had done so, Brownian motion would have jostled it into a new orientation anyway. Howard Berg, working later with the physicist Edward Purcell, worked out the physics of what a cell of that size can and cannot smell. The verdict was unforgiving. At this scale, comparing here to there is not a hard problem. It is close to a physically foreclosed one.
Sit with what that means. The cell is too small to hold a “here” and a “there” apart. It cannot perceive space, in the sense that matters for navigation. And yet it finds the sugar, reliably, and starves if it does not.
So it must be solving the problem some other way.
The solution: sense time instead
In 1972, at the University of Colorado in Boulder, Howard Berg and Douglas Brown published a paper in Nature describing what individual E. coli cells actually do when they swim. The result was possible because of an instrument Berg had built: a three-dimensional tracking microscope that could lock onto a single bacterium and follow it through the water, drawing its path in space. Paul Horowitz, the Harvard physicist better known for teaching a generation of scientists electronics, had helped with the circuitry. The machine was a strange and beautiful thing, an apparatus whose entire purpose was to chase one invisible animal around a drop of water and write down where it went.
What it wrote down was this. The cell does not swim in a purposeful line. It runs, straight and smooth, for about a second. Then it tumbles, a brief chaotic spin lasting about a tenth of a second, which throws it off in a new and essentially random direction. Then it runs again. Run, tumble, run, tumble. Plotted out, the path is a drunkard’s walk, a stagger with no obvious destination.
The discovery was not in the walk. It was in the bias.
When a cell happens, by chance, to be swimming in a direction where conditions are improving, it does not steer harder. It does something subtler: it suppresses the next tumble. It simply keeps going a little longer before it spins. When conditions are flat or getting worse, it tumbles on schedule and tries a new heading. Nothing about the tumble itself is aimed. The new direction is still random. All the cell controls is how long it is willing to keep swimming before it randomizes again.
That is enough. Over many runs and tumbles, a walk with that one bias in it drifts, statistically, up the gradient. The cell arrives at the food without ever knowing which way the food was. The finding Berg and Brown reported has a deflationary elegance that is easy to miss on first reading: the cell never acquires a heading toward the sugar. Its changes of direction were happening anyway, spontaneously, in plain water. All chemotaxis does is edit their timing. In the same year, working with a different method, Robert Macnab and Daniel Koshland reached the same conclusion. Two labs, two approaches, one answer.
Notice the shape of what the cell has done, because it is the shape of everything that follows in this series. It could not perceive space, so it perceived time. It replaced the question which way is better with the question is it getting better. The first question requires a map. The second requires only a memory.
The memory: what a cell remembers, and for how long
Which raises the obvious question. To know whether things are getting better, the cell has to compare now against then. So where is the “then”?
Brown and Berg went after this directly in 1974, exposing cells to concentrations that changed in time rather than in space, and watching how they responded. If the cell’s behavior depended on a remembered past, the shape of the response would reveal how long that memory lasted. Their conclusion was that if the bacterium has a memory, it decays in a few seconds at most. A few seconds is the entire temporal horizon of this organism. Everything it knows about the world, it learned in the last breath.
The molecular story, filled in over the following decade by Julius Adler’s lab and others, turned out to be more elegant than anyone had a right to expect.
Sugar molecules bind to receptor proteins studded in the cell’s membrane. Those receptors are wired to an enzyme called CheA, which, when the receptor is active, attaches phosphate groups to a small protein called CheY. Phosphorylated CheY drifts through the cell to the flagellar motors and changes the odds of which way they spin. Counterclockwise rotation braids the flagella into a bundle that pushes the cell forward, and the cell runs. Clockwise rotation blows the bundle apart, and the cell tumbles. So the entire behavioral repertoire of the organism comes down to a single number: the concentration of one phosphorylated protein, which sets the probability of a coin flip at the motor.
The memory is in what happens next. A second pair of enzymes, CheR and CheB, slowly add and remove methyl groups to those same receptors. Methylation adjusts how active the receptor is at a given level of sugar. It moves, deliberately and with a lag of seconds, toward whatever level of sugar the cell has recently been swimming through. The methylation state is a chemical record of the recent past.
And here is the part worth stopping for.
Because the methylation state tracks the recent past, and because it pushes the receptor’s activity back toward its baseline, the receptor’s output does not report how much sugar is out there. Steep-climb or slow-drift, high background or low, the cell adapts, and the signal returns to where it started. What the receptor actually reports is the difference between the sugar the cell is tasting right now and the sugar its recent history led it to expect.
The cell subtracts its own expectation. What is left, the residual, is what drives behavior.
That is a prediction error. Not a metaphor for one, and not a loose family resemblance. Structurally, it is the thing itself: an internal expectation, formed from recent experience, subtracted from present input, with the difference used to select an action. The bacterium suppresses its tumble not because the world is sweet, but because the world is sweeter than it predicted.
Be precise about the limits, because the differences matter as much as the thread. This is not temporal-difference learning. There is no value function. There is no bootstrapping from one estimate to the next, no discount factor, no credit assigned backward across a chain of decisions. The cell is not learning across its lifetime in any rich sense; its policy is fixed and its “learning” is a two-second sliding baseline. Everything that makes modern reinforcement learning powerful is absent.
But the core is there. The core is a system that acts on the gap between expectation and reality. When we reach Rescorla and Wagner in the 1970s, formalizing animal learning as driven by surprise rather than by reward, and when we reach Sutton’s temporal-difference methods, and when we reach the discovery that dopamine neurons in the mammalian midbrain fire not to reward but to reward prediction error, we will not be watching the invention of a principle. We will be watching its rediscovery, in a substrate that could afford to run it faster. What matters here is where the principle was first implemented, and by what.
It was implemented in a cell, in protein, with no neurons at all. Nobody can put a precise date on when the circuit that does it first appeared. What can be said is the part that matters: bacteria were running loops of this shape for billions of years before there was a nervous system anywhere on Earth to run one faster.
Weber’s law, in something that cannot hear
There is a second consequence of that adaptation machinery, and it is the detail I would put on a poster.
Because the methylation state slides to match the background, the cell ends up responding not to absolute changes in sugar concentration but to fractional ones. A jump from one unit to two produces the same response as a jump from a hundred to two hundred. The cell is sensitive to ratios, not to differences. It performs, in effect, logarithmic sensing. Adler’s group noticed this in the early 1970s and named it for what it plainly was.
They called it Weber’s law.
Ernst Weber was a German physiologist working in the 1830s, and what he found, with weights and human hands, is one of the oldest quantitative results in psychology. The smallest difference a person can detect is not a fixed amount. It is a fixed fraction of the starting stimulus. Hold a light weight and you can feel a small addition. Hold a heavy one and the same addition disappears; you need a proportionally larger increase to notice anything. Loudness, brightness, weight: the same rule, again and again, across the human senses.
And then, a century and a half later, in an organism with no senses in any ordinary meaning of the word, no ears, no eyes, no nerves, no brain, the same law turns up. A cell obeys Weber’s law.
This is exactly the kind of coincidence this series exists to explain, and it is not a coincidence at all. Ask the question the information lens tells us to ask: what problem does logarithmic sensing solve? Any system with limited dynamic range that must operate across a huge span of background conditions faces the same bind. Sugar in a gut varies over orders of magnitude. So does light on a face, from starlight to noon. A detector that measured absolute differences would be saturated at the top of the range and useless at the bottom. A detector that measures ratios stays useful everywhere, because it spends its precious sensitivity on change relative to context rather than on the context itself.
Weber’s law is not a quirk of human perception, and it is not a quirk of bacterial chemistry. It is what the solution looks like when the problem is: allocate a small dynamic range across a world that spans many orders of magnitude. The cell and the cortex are separated by four billion years and every conceivable difference of implementation. They landed on the same rule because they were handed the same problem, and the problem has a shape.
This is worth remembering, because it will keep happening. When we reach the normalization layers that keep modern neural networks trainable, we will be looking at a third landing on the same rule.
The three questions
The instrument this series carries from stop to stop is a set of three questions. Run them on the bacterium.
Which regularity in the world does it latch onto? A specific and rather subtle one: that chemical gradients in water are smooth. Because concentration changes gradually across space, a change measured over time, as the cell swims, is a reliable proxy for a change measured over distance, which the cell cannot measure directly. That fact about the physical world is what makes the whole strategy work. In a world where sugar was distributed in randomly scattered pockets with hard edges, the run-and-tumble bias would buy nothing. The cell’s strategy is not generally intelligent. It is exquisitely fitted to one true and exploitable property of its environment. Every intelligence in this book will be, in its own way, a bet on a regularity.
What does it compress that regularity into? Almost nothing: the methylation state of a few receptor proteins, a chemical record of the last second or two, decaying continuously. Call it a model, because it functions as one, but note how poor it is. It is a single running expectation. It has no structure. It does not represent where the food is, only whether the last moment was an improvement. It cannot be queried, imagined, or replayed. It is a model with no map in it.
What does the model buy? Food, and therefore a few more hours of holding off equilibrium. That is the whole ledger, and it is a brutally direct one. There is no partial credit here, no gradient of usefulness. A cell that models the gradient eats. A cell that does not starves. Prediction, in this organism, is not a cognitive luxury. It is metabolism by other means.
That third answer is the one to hold onto. It is the point where this chapter touches Schrödinger, who argued that a living thing survives only by continuously importing order and exporting disorder, running against the universal slide into equilibrium for as long as it can pay for it. To pay, it has to act. To act well rather than at random, it has to predict which actions pay. To predict, it has to model. The bacterium climbing its gradient is not doing something adjacent to staying alive. It is doing staying alive, described at a different level.
When there is nothing to predict
Every strategy in this book is a bet on a regularity. Which raises the question that ends this chapter: what does a system do when there is no regularity to bet on?
Bacteria face this too, and their answer is the strangest thing in this story.
In 1944, in The Lancet, an Irish physician named Joseph Bigger reported something odd about penicillin. He was killing staphylococcal cultures with it, and the drug worked, mostly. It reliably destroyed the overwhelming majority of the bacteria. But not all of them. A tiny remnant survived, and when he regrew that remnant and hit it with penicillin again, the new population died just as readily as the first. The survivors were not resistant. Their descendants had no special immunity. They had simply, somehow, not been killed. Bigger called them persisters.
Persisters were studied on and off for the next six decades, but the mechanism stayed dark. It took until 2004, and a technology Bigger could not have imagined, to see what those survivors actually were. Nathalie Balaban and colleagues used microfluidic devices to watch individual E. coli cells over time, and reported in Science what the population was doing. In a genetically identical culture, a small fraction of cells spontaneously switch into a dormant, slow-growing state. They are not sick and they are not mutants. They are the same organism, running a different program. And because many antibiotics kill by attacking the machinery of growth, a cell that has stopped growing is a cell the drug cannot get a grip on. When the antibiotic clears, the dormant cells wake up and rebuild the population.
Read that as a strategy and it is startling. The population is deliberately keeping a fraction of its members useless. Persisters grow slowly. In good times they are pure cost, a tax the colony pays every generation. What the tax buys is insurance against a catastrophe that the cell has no way whatsoever to see coming.
And that is the point. An antibiotic does not announce itself with a gradient. There is no smooth chemical slope leading gently up to a lethal dose, no signal in the water a few seconds beforehand that a cell could adapt to and act on. The regularity that chemotaxis exploits, the world changes smoothly, so recent history predicts the near future, simply does not hold for this class of disaster. The prediction machinery has nothing to grip.
So the organism switches strategies. When you cannot predict, do not predict. Randomize, and make sure that whatever happens, some fraction of you was doing the other thing.
This is bet-hedging, and it is the rawest form of a trade-off that will follow us through this entire series. The cell that keeps swimming up the gradient is exploiting: it is committing to its model of the world and cashing it in. The cell that goes dormant for no reason is exploring, or more precisely, it is hedging: paying a certain cost now against an uncertain catastrophe later. Any system that has to learn by acting inside a world it does not fully know is forced to strike this balance somewhere, and here it is at the very bottom of the tree of life, implemented not as a decision but as a coin flip built into a genetic circuit.
Notice where the “model” lives in this case. No individual persister has decided anything. No cell perceives the threat or reasons about the risk. The knowledge that catastrophes happen, and that a hedge is worth its cost, is not in any cell’s memory at all. It is in the genome, written there over evolutionary time by the simple fact that lineages which hedged survived the disasters and lineages which did not, did not.
Which gives us the second and much older memory in this chapter. The methylation state is a model of the last two seconds, held in the individual, and it dies with the cell. The genome is a model of the last four billion years, held in the lineage, and it is the accumulated record of what worked. Both are compressions of the world into a form that guides action. They differ in what they were compressed from, and in how fast they can change. One is written by experience over seconds. The other is written by selection over aeons, and no individual can edit it.
That is not a metaphor either. It is the reason evolution and learning keep looking like the same process viewed at different speeds, a resemblance we will meet again and be careful with.
What the cell could not do
The loop was complete before neurons. That is the finding of this chapter, and it should reframe what comes next. Sense the world, compare it against an expectation formed from recent experience, act on the difference, and let the lineage carry the slower lessons. All of it, in a cell, in protein, with no nervous system.
Which means that the neuron, when it arrives, is not the invention of intelligence. It is a change of substrate.
So the useful question is the one the immunity statement demands: not whether the bacterium is doing what we do, but what it could not do, and therefore what remained to be invented.
It could not remember for longer than a few seconds. Its horizon is the width of a breath, and any regularity in the world that unfolds over minutes or days is invisible to it.
It could not represent where anything was. It has an expectation, but no map. It cannot hold the layout of its world in any form, which means it cannot plan, cannot take a detour, cannot go around.
It could not simulate. It cannot ask what would happen if it swam the other way, because it has nothing in which to run the experiment except the world itself, at full cost, in real time.
It could not model another cell. It has no representation of an agent other than itself, and therefore no way to predict, cooperate with, or deceive one.
And it could not tell anyone. Whatever this individual cell learned in its lifetime dies with it. The only channel from one generation to the next runs through the genome, which is slow, lossy, and cannot carry a specific fact about this particular drop of water.
Five absences. Each one is a chapter of this series, and each one will be filled by a solution that costs more, runs slower per unit of thing done, and buys a longer horizon. Memory that outlasts the moment. A map instead of a gradient. A simulation you can run before you act. A model of other minds. A channel to transmit what you learned to someone who was not there.
That is the road from here. It starts, as everything in this history does, with a cell in a drop of water, comparing now to then, and betting its life on the difference.
The formal machinery behind this chapter, prediction error and the algorithms built on it, gets taken apart properly in The Journey of RL.
This is Part 1 of An Unfinished History of Intelligence, a series that follows a single thread, the one problem life, mind, and machine all solve, from the first living cell to the machines we are building now. Each installment stands on its own and connects to the deeper technical series across Robonaissance.
superb essay....looking forward to reading the whole series...
This is wonderful. I'm reading this series as I use codex to build an application and I keep ignoring codex's "i'm finished" pings to read and reread sections multiple times. It takes some really interesting writing to get me to ignore the next dev prompt. :)