The Thermodynamics of Intelligence, Part 3: The Speed Limit Nobody Obeys
Cool metal slowly and atoms find order. Cool it fast and the flaws freeze in. There is a theorem for how slow is slow enough. Nobody can afford it. The field advanced by cheating.
The Blacksmith
The most physical learning algorithm ever written was published in 1985, has an exact convergence theorem behind it, and is too slow to use. Everything the field built afterward came from violating the conditions that theorem lays down, deliberately, by people who could name the theorem they were breaking.
That is the argument. It begins at a forge, because the algorithm did.
The metal comes out of the fire glowing, and what happens next is a decision.
Plunge it into the water barrel and it hardens in a second, steam roaring off the surface. Or leave it in the dying forge and let the heat bleed away over hours, sometimes overnight, the color fading through orange and dull red into nothing. Same steel, same starting temperature, two different materials at the end.
The difference is what the atoms had time to do. Hot metal is a crowd with energy to move; atoms shuffle, slide past dislocations, and drift toward positions that suit them. Cooling drains the energy that makes moving possible. Cool slowly and the crowd keeps rearranging as it settles, grains growing large and regular, internal stresses working themselves out, the whole mass easing toward an arrangement of low energy. Quench it and none of that happens. The atoms are frozen wherever they stood at the moment the heat left, defects and stresses locked in place, the material trapped in a configuration it would never have chosen given time.
Smiths have known this for as long as there have been smiths, and have a word for the slow version. Annealing.
Strip out the metallurgy and what the slow cool accomplishes is this. There is a landscape of possible atomic arrangements, each with an energy. There is a system trying to find a low point in that landscape. At high temperature it wanders freely and can climb out of anywhere. As the temperature drops its wandering shrinks, and it becomes progressively more committed to wherever it currently is. Lower the temperature slowly enough and the system commits only after it has found somewhere good. Lower it too fast and it commits early, to whatever mediocre spot it happened to occupy.
That is an optimization algorithm. A furnace is running it, atoms are executing it, and the smith is reading its output by eye.
Part 2 of this series ended on a question. Noise in a Hopfield network turned out to be a way of telling a deep valley from a shallow one, because escaping is the only test of depth available from inside. If noise can sort the memories a network already holds, can it build them? This installment is the answer, and the answer arrives in two stages separated by thirty years, both of them lifted directly out of the forge.
