This is Episode 18 of Reading the Frontier. What the frontier’s evidence shows, and what we think it means.
A language model learned to write by reading more text than any person could finish in thousands of lifetimes, and almost none of that text was written for it. People wrote it for each other, and the model got it free. Nobody has assembled anything like that for a robot, because there is no internet of robot experience. Every hour a robot spends learning to fold a cloth or open a drawer had to be paid for: a machine, a person steering it through the motion, someone resetting the scene after each try. Meanwhile people film themselves doing exactly these things all the time, chopping onions, wiping counters, stacking cups, and post the videos by the million.
So the obvious question is whether a robot can learn from us. One result posted in October 2026 suggests it can, and also shows the catch. A robot can borrow experience from a body that is not its own only through a translation of what that body did into something the robot could do. The quality of the translation decides what the borrowed experience is worth, much more than the number of hours. A modest amount of the robot’s own experience still holds the whole thing in place.
Why a video is not a lesson
Watch someone pick up a mug in a video and you learn a great deal: where to reach, roughly how to grip, that the handle matters. A robot watching the same clip learns almost nothing it can use directly, and there are two separate reasons.
The first reason is that a video shows what happened, not what was commanded. Robot training data comes in pairs: what the robot saw, and the action it took next, written as motor instructions. A clip of a person holds the seeing and none of the instructions. Nobody recorded the force in the wrist or the angle of each knuckle. A robot cannot imitate a command it was never shown.
The second reason is that the body is wrong. Suppose you could recover every joint of the human hand from the footage. A robot’s body is not built like a human hand. A perfect record of a human hand is a perfect record of a body the robot does not have. You can see the gap for yourself: pick up a pen, then try to describe the motion a pair of tongs would have to make to do the same thing. Most of what your fingers did has no tongs equivalent.
So human video needs two things before it is worth anything to a robot. It needs labels, and the labels have to be in a language the robot’s body can speak.
Translating change, not anatomy
The team behind the result, researchers at JD’s Joy Future Academy and several Chinese universities, attacked both problems with a single idea. Instead of asking what the person’s hand did, their system asks what changed between two consecutive frames.
Here is the step that matters. Show a model two frames, a cup on the table and then the cup slightly lifted. Ask it to compress the difference into a short code, a brief list of numbers. To check the code, give another part of the model only the code and the first frame, and ask it to predict the second. The first frame says where things started, so the code has to carry what changed. A good code is one that makes the prediction come out right. The code is not a written instruction such as “lift the cup,” and it never names a finger or a joint. It is a learned representation of the change in the scene: something moved up, something rotated, a drawer slid. Run this across every pair of consecutive frames in thousands of clips and the silent footage acquires labels. Each moment is tagged with what was done to the world, written in a vocabulary invented from the video itself. The team’s labelling model learned this vocabulary from 23,300 hours of people’s first-person video drawn from six sources.
The vocabulary is the translator. A description of what changed in the world is something a robot’s body can aim for, even though a description of what a hand did is not.
Training then runs in three stages, all on the same robot platform, the AgiBot G1. First, the robot’s model learns from human video labelled this way, a cleaned 5,100-hour subset kept to clips with visible hands and real change, predicting which change comes next from what it sees. Second, it is aligned on 250 hours of the robot’s own data, where the robot learns which of its own motor commands bring about each kind of change; those 250 hours are the robot’s modest store of its own experience. Third, it is trained on the specific task, from a separate and much smaller set of 50 demonstrations per task. The human video teaches what tends to happen next in a scene, the robot’s own hours teach how this body makes it happen, and the task data teaches what to do here.
What more hours bought
The test that makes the result interesting holds the robot’s own data fixed and varies only the human video. If more footage raises success while nothing else about the robot’s experience changes, the footage is carrying something real.
It did, on three real-robot tasks, one with a mouse pad, one with headphones and one with a cup, each tried 20 times per setting. As the human video grew from 1,000 hours to 3,000 and then 5,000, in nested subsets of the cleaned footage, success across the 60 trials rose from 63.3% to 80.0% to 85.0%, against 55.0% with no human video at all. Those are combined scores for the three tasks, and single tasks swung widely enough (the cup task went 70%, 95%, 75%) that the trend carries more weight than any one number.
More pretraining of any kind tends to help, so the team checked whether the video was simply more training. They gave a version of the robot extra training on its own data, with the computing budget matched to what the human video consumed. That version reached 53.3%, no better than the 55.0% with no human video, against 85.0% for the video-trained one. The gain came from what was in the footage, not from the extra hours of computation.
The counter-case
The most telling result is a failure. The team took the same human video and labelled it a different, more intuitive way: by tracking the person’s hand, its position and pose, frame by frame. That is the label most people would reach for first. If the robot is learning from a hand, why not describe the hand?
Hand-tracking labels helped through the middle of the range, then fell back: success went from 58.3% at 1,000 hours to 75.0% at 3,000, close to the full method’s 80.0% there, then dropped to 55.0% at 5,000, the same as using no human video at all. A second alternative, codes trained only to redraw the next frame’s pixels rather than to isolate what changed, did keep climbing with more video, to 61.7%, 75.0% and 76.7%, but stayed below the full method’s 85.0%.
The footage was identical across these comparisons. Only the translation differed. The authors suggest, without testing it, that hand tracking gets noisier as footage brings in camera motion, hands hidden behind objects and changed viewpoints, so more of it adds more noise. My own reading adds a second suspicion: labels that describe the human body pass along the second problem, the wrong body, unsolved, so a small amount of hand data works like a rough hint about where to reach, while a large amount starts teaching the model to care about human anatomy, which the robot’s own hours then have to unlearn. Labels that describe the change in the scene kept paying as the hours grew, which fits the idea that the change is the part both bodies share. Think again of the person stacking cups: tracking every bend of their fingers describes a hand the robot does not have, while capturing how a cup moves into place describes something the robot’s own body might do.
That is why “just collect more video” is the wrong summary of the result. More video helped consistently only when the labels followed what changed between frames. Translated badly, a large library of footage was worth about as much as none.
Where it held and where it tied
The team also added four harder tasks to the three original ones: paper-ball disposal, pen grasping, eraser grasping and multi-stage book stowing, each trained from 200 demonstrations. This was a single test of each task, run 20 times in conditions the robot had seen during training and 20 times under each of three shifts: a moved object, changed lighting and a new background. In seen conditions, the video-trained robot and a strong baseline trained without this method tied at 85.0%, while under the shifts the video-trained robot led, 75.0% to 50.0% for object position, 70.0% to 45.0% for lighting and 70.0% to 40.0% for background. A plain imitation policy managed 60.0% in seen conditions and fell to between 15.0% and 20.0% under the shifts.
That split is what you would expect if the footage supplies breadth rather than skill. When the robot has already practised the exact situation, its own experience is enough, and the borrowed hours add little. When the situation departs from anything in its own data, the borrowed hours are the only experience it has of that kind of scene. Human video, on this evidence, does less to make a robot better at what it has done than to make it less surprised by what it has not.
The evidence has to be weighed for what it is: one team’s own report, seven tasks on one robot platform and 20 trials per task per condition, which the authors themselves say is too little to claim the robot would hold up in deployment. Twenty trials means each success or failure moves a single task’s score by five percentage points, so the large gaps carry weight and the small ones may not. The shape of the result is more trustworthy than any single number in it. The shape is that more footage helps, translated footage helps more, and the robot’s own data never drops out.
Where the experience will come from
The question these robots face is the one language models never had to ask. Text was already in the form the model needed. Usable robot experience is scarce at anything like that scale, and the experience that does exist in abundance, us, on camera, is recorded in the wrong body with the instructions missing.
I think this points to a plausible division of labour for how robots will learn. Most of what a robot knows before it starts a task will probably come from watching people, because people have filmed far more activity than robots have carried out. What makes that footage usable is a translation into the changes a task makes in the world, and the field’s effort will shift from gathering footage to building better translators. The robot’s own hours will shrink as a share of what it learns without disappearing, because, as the authors note, the invented labels are not guaranteed to line up with the joints a robot can actually move, so tying them to a real body still takes the robot’s own data. A robot that has never moved cannot know what its own body does to a cup, however many hands it has watched.
So where will robots get their experience? Mostly from us, through a translator, on one team’s evidence from one machine. The video has not yet shown it can replace the robot’s own hours, which tie the translation to a real body.
This is Episode 18 of Reading the Frontier. What the frontier’s evidence shows, and what we think it means.


