This is Episode 3 of Memory, Forgetting, and the Self in the Machine: how intelligence reaches across time when it cannot carry the whole past with it.
Freeze a game of Pong on any frame. Two paddles, a ball, a score. Now say where the ball is going. You cannot, and neither could a machine shown the same picture. Matthew Hausknecht and Peter Stone, writing in 2015 about agents that learn to play Atari games, put it flatly: “the current screen only reveals the location of the paddles and the ball, but not the velocity of the ball.” Then they explain why that matters: “Knowing the direction of travel of the ball is a crucial component for determining the best paddle location.” The frame is accurate. Everything on it is where the frame says it is. It is also not enough to play the game.
What the sensor shows at this moment is an observation. What the next move depends on is the state. In Pong the two come apart on every single frame, because the ball’s direction is a fact about the last few moments that no one moment contains. When the present input falls short like this, a system that acts well has to carry something forward from its past. What it carries is never the whole past. It is a summary, and the summary is chosen for what it has to predict. That choice decides what a later moment has to work with. Anything the summary leaves out, the later moment cannot act on.
Four screens deep
The system Hausknecht and Stone were responding to was DQN, the agent Volodymyr Mnih and colleagues described in Nature in 2015. DQN learned to play dozens of Atari games from the screen, and its authors had already met the frozen-frame problem. In the paper’s Methods section they write: “Because the agent only observes the current screen, the task is partially observed and many emulator states are perceptually aliased.” In other words, the current screen alone cannot tell the agent its full situation. Perceptually aliased is a precise phrase for an ordinary trouble. Two different situations, such as a ball heading toward your paddle and a ball heading away from it, produce the same picture. A system that sees only the picture has to treat them as one situation, and if they call for different moves, it will get one of them wrong.
Their remedy was simple. Instead of one screen, the network takes in the most recent frames stacked together, “in which m = 4, although the algorithm is robust to different values of m (for example, 3 or 5).” Lay four frames side by side and the ball’s motion becomes visible as a change in position. Direction stops being something to remember and becomes something to see. The paper does not call the stack a memory. It is an input that happens to be wider than one moment.
The width has a hard edge, and Hausknecht and Stone describe exactly where it sits. Because DQN “is trained using an input consisting of the last four states the agent has encountered,” they write, it “will be unable to master games that require the player to remember events more distant than four screens in the past.” For DQN, the fifth screen back is simply not in the input.
