Free Energy: The Physics Beneath AI
Intelligence may not require knowing where to go. It may require only a landscape in which some directions are downhill.
Put a ball on a slope and let go. It moves.
The ball has no model of the valley. It does not calculate a trajectory, enumerate possible destinations, or represent the state toward which it is heading. At every instant, the physics is local. Yet the result looks global: the ball finds lower ground.
This is one of the oldest tricks in nature. A gas relaxes toward equilibrium. A crystal assembles. A protein folds. A chemical reaction proceeds when conditions favor it. Enormous spaces of possible configurations are navigated without anything searching them one by one. None of these systems knows where it is going. What does the work is the shape of the space they move through: a landscape where, from any point, some directions lead down. Local physics is enough to follow them, and free energy is one of the quantities that defines which way down is.
The idea began in nineteenth-century thermodynamics, in the work of Hermann von Helmholtz and Josiah Willard Gibbs. It passed through statistical mechanics, where Ludwig Boltzmann connected energy to probability. Then it appeared again in Bayesian inference, machine learning, energy-based models, variational methods, and theories of biological intelligence.
The equations are not always describing the same physical quantity. That distinction matters. But the computational idea underneath them is remarkably persistent: assign a single number to every possible configuration, turning a difficult global problem into a landscape, then let local dynamics do the search. That may be one of the deepest connections between physics and AI.
Conservation says nothing about direction
Energy is conserved. Use some, and it does not disappear. It changes form.
That makes energy fundamental, but it also creates a problem. If the total amount of energy in an isolated system does not simply vanish, why do physical processes have preferred directions? A hot cup of coffee cools; it does not spontaneously draw heat from the room and become hotter. A gas released into a box spreads out; it does not ordinarily gather itself into one corner again. The total energy can remain the same while the system loses something else: its capacity to produce directed change.
Thermodynamics captures this through free energy. The simplest version, Helmholtz free energy, is written
F = U − TS
where U is the system’s internal energy, which is what the word energy has meant everywhere above: the total energy the system has stored, in molecular motion, in bonds, in interactions between particles. T is temperature and S is entropy.
The reason this quantity matters is that systems move toward lower values of it. Left alone at a fixed temperature, a system settles where F is smallest. That is the sense in which free energy defines which way is down.
Read literally, the formula makes free energy look like a portion of the internal energy: start with U, subtract TS, keep the rest. That is a reasonable working intuition and close to what free means here. The TS term is the part entropy holds back, unavailable for doing work, and the hotter things get the more of it is held.
The size of that deduction depends on temperature, which is a property of the surroundings rather than something stored inside the system. The same system, with exactly the same U, placed somewhere colder, has a different F. So the two quantities answer different questions. U asks how much energy is in there. F asks how much of it the current conditions will let you use.
Entropy is the term that makes this interesting, and what it does is count arrangements. Flip a hundred coins. There is exactly one way for all of them to land heads, and roughly 10^29 ways for exactly half of them to. Nothing pushes the coins toward a fifty-fifty split; no force prefers it. That outcome simply has overwhelmingly more ways of happening, and the counting alone is enough to make it near-certain.
Physical systems work the same way. Entropy counts how many distinct microscopic arrangements produce the same overall state, and the formula shows why that matters. F gets smaller when U is small, and F also gets smaller when S is large. So a state can earn a low free energy in either of two quite different ways: by being low in energy, or by having far more arrangements available. The two routes often pull in opposite directions.
Water and ice are the familiar case. Ice has lower energy, its molecules locked into a lattice by bonds that cost energy to break. Liquid water has vastly higher entropy, since the same molecules can be arranged in an astronomical number of ways while still being water. Which one nature chooses depends on temperature, and the formula says exactly how. When T is small, the TS term barely counts, the low-energy state wins, and water freezes. When T is large, the TS term dominates, the high-multiplicity state wins, and ice melts. At exactly zero degrees Celsius the two terms balance, which is why that is where the change happens.
That is what temperature is doing in the formula. It sets how much the system cares about having options. Minimizing free energy is therefore not the same as minimizing energy, and not the same as maximizing entropy. It is finding where the trade between them settles, given how hot things are. Nature has converted a competition between energy and multiplicity into a landscape.
Boltzmann’s bridge
The connection becomes deeper in statistical mechanics. A macroscopic object looks simple: a temperature, a pressure, a volume. Microscopically, however, it may contain an astronomical number of particles, each contributing to an even more astronomical configuration space. There is no practical sense in which nature can enumerate these possibilities, so statistical mechanics describes them probabilistically instead.
Boltzmann’s contribution was to make that probability explicit. The chance of finding a system in some configuration with energy E, at temperature T, is proportional to
p ∝ e^(−E / kT)
where k is Boltzmann’s constant. Lower-energy configurations are favored; temperature determines how strongly the system is willing to occupy higher-energy alternatives. This creates a remarkable bridge: energy and probability become two descriptions of the same landscape. Low energy corresponds to high probability, high energy to low probability, and that single equation would later become extremely useful in machine learning.
If you can assign an energy to every possible configuration, you can implicitly describe which configurations are plausible without beginning by assigning an explicit normalized probability to every one of them. That is the basic intuition behind energy-based models, going back to the Boltzmann machine that Geoffrey Hinton and Terrence Sejnowski built in the mid-1980s using exactly this distribution. Instead of asking what probability a configuration should have, the model asks how compatible that configuration is with the structure it has learned. Give plausible configurations low energy, implausible ones high energy, and learning becomes landscape construction. Inference becomes movement through that landscape.
A model does not need the answer
Suppose an AI observes something but cannot directly see what caused it. The visible data may have been produced by countless hidden configurations, and Bayesian inference asks which of those hidden states are plausible given what was observed. In simple problems, that question can be answered exactly. In interesting ones, it often cannot: the space of possible explanations becomes too large, and calculating the full distribution over them is itself the impossible problem.
Variational inference, developed through the 1990s by researchers including Geoffrey Hinton, Michael Jordan, and their collaborators, makes a peculiar move. It stops trying to calculate the answer. Instead, it creates a simpler distribution of beliefs that the machine can represent, then gradually changes that distribution until it behaves as much as possible like the inaccessible one. Inference becomes optimization: one pressure forces the belief to explain the observation, another prevents it from becoming needlessly strange, brittle, or detached from what was plausible beforehand. The result is conventionally described using variational free energy.
This is not Helmholtz free energy hiding inside a neural network. The quantities have different meanings. But the mathematical architecture has survived the journey from physics into inference. There is a space of possible states, a scalar quantity that ranks them, and a dynamics that moves toward better regions without ever calculating the final answer directly. The belief does not solve for the posterior. It relaxes toward it. A problem that looked like calculation has been converted into motion.
Inference can be relaxation
Normally, we imagine reasoning as the production of an answer: question in, computation happens, answer out. Variational inference suggests another picture. Begin with an imperfect distribution of beliefs. Construct an objective landscape over those beliefs. Then repeatedly modify them so that the objective improves, until eventually the beliefs settle into a configuration that approximates the posterior. The answer was never explicitly calculated in one step. It was approached through relaxation.
This is a surprisingly physical picture of inference. A belief can occupy a bad configuration in much the same abstract sense that a physical system can occupy a high-free-energy configuration. New evidence changes the landscape, the distribution moves, and a new equilibrium becomes preferable. The inference problem has become something a dynamical system can solve.
The same trick appears across AI
Once you see this pattern, it becomes difficult not to see it everywhere.
A neural network does not receive its final weights; training defines an objective and lets optimization discover them. An energy-based model does not list every valid configuration; it learns a surface on which valid configurations lie lower. A Hopfield network, the model John Hopfield introduced in 1982 by way of the same statistical mechanics as the Boltzmann machine, stores memories as attractors: give it a corrupted pattern and its dynamics settle into a nearby basin. A diffusion model begins with noise and repeatedly follows a learned field toward regions compatible with the data distribution. A variational autoencoder does not assign each observation a single hidden explanation; it learns a distribution over latent explanations while balancing reconstruction against a prior.
These systems are technically different, and calling all of them free-energy minimizers would erase important distinctions. But they share an architectural move: a hard global problem is transformed into local dynamics on a designed landscape. This is more general than gradient descent. It is a way of outsourcing computation to geometry.
The objective is the hidden architecture
This also changes how we should think about objectives. A loss function can look like a bookkeeping device, a number printed at the end of a training step telling us whether the model improved. But the number is not merely measuring the process. It is shaping the space in which the process occurs.
Change the objective and the directions that count as downhill change with it. Change the regularization and the valleys reshape. Change the data and entire basins can appear or disappear. Change temperature-like parameters in probabilistic systems and narrow optima may give way to broad families of states. The objective therefore acts less like a score and more like a geometry imposed on possibility, which is why apparently small changes to an AI objective can produce qualitatively different behavior. The optimizer follows the landscape it is given. It has no independent concept of where it ought to arrive. Physics gives the same warning: minimization is powerful only because the quantity being minimized encodes the right constraints. A system can descend perfectly and still descend the wrong landscape.
Local minima are not accidents
Landscapes also explain why intelligent systems can get stuck. A ball descending a mountain does not necessarily reach the lowest point on Earth; it reaches whatever basin its trajectory allows. Physical systems encounter metastable states, configurations that are locally stable even though lower-free-energy states exist elsewhere. Glass is a classic example: its microscopic structure can become trapped away from equilibrium because reaching a better configuration requires crossing barriers.
Optimization has the same problem. A learning system can find a configuration that is locally good without being globally best, and noise, temperature, initialization, momentum, stochasticity and architecture all influence which barriers can be crossed and which basin is eventually occupied. This suggests a richer question than simply where the minima are: what trajectories through the landscape are dynamically accessible? Two systems can share an objective and still learn differently because their dynamics differ. The landscape defines possibility. The dynamics define reachability. Intelligence may require both.
A world model as a landscape
A world model is usually described as a machine that predicts what happens next: given the current state and perhaps an action, estimate the future. But prediction is only one way to represent a world. Another is to learn the structure of possible states themselves, which configurations belong together, which transitions are easy or impossible, which observations can be explained by the same hidden cause. In that picture, a useful world model is not merely a simulator. It is a geometry.
An agent navigating the physical world then faces multiple overlapping landscapes at once: energetic costs, uncertainty, expected reward, risk, social constraints, learned preferences, and the probability of future states. Planning can sometimes be reformulated as inference over trajectories. Control can sometimes be reformulated as making preferred trajectories probable. The boundaries between prediction, inference, optimization and action begin to blur, not because they are identical, but because they can sometimes be expressed as different ways of moving probability mass through structured spaces.
The dangerous leap
There is a temptation here. If physical systems minimize free energy, and inference can minimize variational free energy, and brains perform inference, perhaps intelligence itself is simply free-energy minimization. That claim is much stronger than the mathematics warrants.
A scalar objective does not explain where the objective came from. A minimum does not explain why that minimum is useful. A landscape does not explain how its coordinates were chosen. Optimization does not by itself produce agency. The more important question lies one level above minimization: who or what shapes the landscape? In thermodynamics, physical law and boundary conditions do. In machine learning, the answer includes model architecture, training data, objective functions, priors, environments and human choices. In organisms, evolution, development, physiology and experience all contribute. The descent may be automatic. The landscape is not.
This distinction matters increasingly as AI systems become more autonomous. Giving a system a better optimizer makes it better at descending. Giving it a better world model lets it perceive more of the terrain. Giving it longer horizons lets it follow paths across larger regions. None of those facts alone tells us whether the valleys correspond to outcomes we actually want.
Intelligence as controlled disequilibrium
There is one final complication. Living systems do not simply reach equilibrium and stop. A dead organism equilibrates very efficiently; a living one continuously consumes energy to maintain itself far from equilibrium, preserving gradients, repairing structures, predicting disturbances, acting on the environment, keeping certain variables inside narrow viable ranges.
Intelligence may therefore be less about minimizing one free energy once than about continually constructing, escaping and reshaping landscapes while remaining far from thermodynamic equilibrium. The organism descends one landscape so that it can continue existing in another. An intelligent agent may do something similar: it reduces uncertainty here to preserve options there, accepts immediate cost to reach a better future state, deliberately climbs local hills because its model predicts a deeper valley beyond them. Once a system can model the landscape itself, simple descent becomes planning. Once it can model how its actions change the landscape, planning becomes agency. And once it can alter the objectives that define what counts as downhill, a much harder problem begins.
The old trick
Free energy began as an answer to a physical question: given the constraints, which changes can happen spontaneously? AI keeps rediscovering variations of the same computational move. Do not enumerate every possibility. Do not calculate the global answer directly if you do not have to. Construct a landscape, define what counts as downhill, and let local dynamics perform the computation.
A protein can fold without knowing its final shape. A Bayesian approximation can approach a posterior it cannot calculate. A neural network can find parameters nobody could specify by hand. A world model may eventually allow an agent to navigate futures too numerous to enumerate. The intelligence is not necessarily in knowing the destination. Sometimes it is in having the right terrain.
And that leaves AI with a question thermodynamics never had to answer: what happens when the thing descending the landscape learns how to reshape the landscape itself?
This article paints the wide picture. For a more systematic treatment of these correspondences, one at a time, see The Thermodynamics of Intelligence, a nine-part series examining exactly how far each one holds.