"It is comparatively easy to make computers exhibit adult level performance on intelligence tests or playing checkers, and difficult or impossible to give them the skills of a one-year-old when it comes to perception and mobility."
— Hans Moravec, Mind Children, 1988
In 2023, Google DeepMind published a study showing that their RT-1 robot had learned to perform over 700 manipulation tasks by watching 130,000 human demonstrations. The demonstrations had been collected using 13 robots over 17 months in an office kitchen environment. Seventeen months. Thirteen robots. One kitchen.
For context: GPT-3 was trained on roughly 300 billion tokens of text scraped from the internet in weeks. Sora was trained on millions of hours of video. The entire written and filmed record of human civilization was available as training data. But robot data, the continuous joint-angle signals and force measurements that describe how a physical body interacts with the physical world, does not exist on the internet. Every trajectory must be collected by a human teleoperating a physical robot, one demonstration at a time.
This is the defining constraint of the Roboticist’s Road. The physicist can simulate worlds from equations. The cinematographer can learn from the planet’s video archives. The robot has almost nothing. And what little it has took months of painstaking labor to collect.
From Kitchen to Knowledge
The breakthrough idea was to borrow. Not robot data, which remained scarce, but knowledge.
In July 2023, Google DeepMind released RT-2, the first vision-language-action model. The recipe was simple in concept and radical in implication: take a vision-language model already trained on billions of images and text from the web, and fine-tune it on robot demonstration data. Represent robot actions as text tokens. Treat motor commands as just another language.
The results were striking. RT-2 could interpret commands it had never seen in robot training data. Asked to pick up an object that could serve as an improvised hammer, it chose a rock. Asked to identify which drink would help a tired person, it selected an energy drink. These were not behaviors learned from robot demonstrations. They were inherited from the vision-language model’s knowledge of the world: knowledge about what rocks are, what hammers do, what tiredness means. On novel, unseen tasks, RT-2’s success rate nearly doubled compared to its predecessor, from 32% to 62%.
The principle was powerful. The web contains vast knowledge about objects, their properties, their relationships, their uses. A model pre-trained on this knowledge and then fine-tuned on robot data could transfer that semantic understanding into physical action. The robot did not need to learn from scratch that a Coca-Cola can is graspable. It inherited that knowledge from the internet.
The Generalist Dream
If RT-2 proved the principle, Physical Intelligence’s π0 pushed it toward a vision.
In October 2024, a team led by Sergey Levine and Chelsea Finn, both veterans of the effort to make robots learn from data rather than from hand-coded rules, unveiled π0: a generalist robot policy trained on over 10,000 hours of demonstration data from seven different robot platforms and 68 distinct tasks. The model could fold laundry from a hamper, assemble cardboard boxes, clear a table, and stack eggs. It could do these things on robots it had never controlled during training, with objects it had never seen, in configurations that differed from its demonstrations.
This was not a laboratory curiosity. Physical Intelligence, founded in 2023 and valued at $2.4 billion by early 2025, was making an explicit bet: that generalist robot policies would do for physical manipulation what large language models had done for text. Just as GPT could write code, summarize documents, and answer questions because it had been trained on enough diverse text, π0 aimed to fold shirts, load dishwashers, and bag groceries because it had been trained on enough diverse robot experience.
The architecture reflected this ambition. π0 was built on a pre-trained vision-language model, inheriting its semantic knowledge. But instead of generating text tokens for actions, it used flow matching to produce smooth, continuous motor commands at 50 Hz. The distinction mattered. Robot actions are not discrete symbols like words. They are continuous trajectories through joint space, and representing them as language tokens was a lossy approximation. Flow matching treated actions as what they are: continuous signals that must be executed in real time.
By April 2025, the team had released π0.5, which could control a mobile manipulator to clean up kitchens and bedrooms in homes it had never seen. The tasks took 10 to 15 minutes. The robot made mistakes. It sometimes opened and closed the same drawer repeatedly, or failed to grasp an unfamiliar handle. But it was operating in genuinely novel environments, figuring out what needed to be done and doing it, without any environment-specific training. Researchers at the University of Pennsylvania who tested π0 independently reported that even at a 20 to 50 percent success rate on simple tasks out of the box, the result marked a major leap: for the first time, you could download someone else’s robot controller, load it onto your own hardware, and watch it do something useful.
The Misalignment
Then, in December 2025, NVIDIA’s Jim Fan published a critique that struck at the foundation of the entire VLA approach.
Fan’s argument was structural. VLA models graft an action module onto a pre-trained vision-language model. But vision-language models are optimized for benchmarks like visual question answering. This creates two problems. First, most of the parameters in these models serve language and knowledge, not physics. The billions of weights that allow a VLM to recognize a Coca-Cola can and describe its properties are not the same weights needed to predict what happens when you tip it. Second, the visual encoders in these models are actively trained to discard low-level details, because high-level understanding is all that question answering requires. But for a robot performing dexterous manipulation, low-level details are everything: the exact position of a fingertip, the angle of approach, the texture of a surface.
The implication was sharp. If the pre-training objective is misaligned with the needs of robotic control, then scaling up the VLM backbone will not proportionally improve the robot’s physical capabilities. More parameters will encode more knowledge about the world’s semantic structure. They will not encode more knowledge about its physical dynamics. Fan argued that video world models, which learn temporal dynamics and physical regularities from pixels, are a more natural pre-training objective for robots than language models trained to answer questions about images.
This critique illuminates a tension at the heart of the Roboticist’s Road. The VLA approach works because it transfers knowledge. RT-2 succeeded precisely because it knew what rocks and hammers and energy drinks were. π0 succeeded because it understood what “clean the kitchen” meant. But knowledge about what things are is different from knowledge about how things behave. Recognizing a glass and predicting what happens when you flick it are different computational problems, and a model optimized for the first may be poorly suited for the second.
The Hard Problem of Contact
The frontier where this distinction matters most is contact.
Every other road to world models can avoid or approximate contact dynamics. The physicist’s simulator can model contact, but only by writing increasingly complex equations that still fail to capture deformable materials, granular media, or the subtle interplay of friction and inertia that governs everyday manipulation. The cinematographer’s video model can generate convincing images of hands grasping cups, but the moment of contact, where force transfers, objects deform, and physics becomes three-dimensional rather than visual, is precisely where video models break down.
The robot cannot avoid contact. Contact is the point. A robot that picks up a cup must model the friction between its fingers and the ceramic, the weight distribution that determines whether the cup tips, the force required to lift without crushing. A robot that folds laundry must model the way fabric drapes, bunches, slides, and catches. These are not edge cases. They are the central problem.
This is why robotics imposes the hardest test on world models. A video model can fake gravity by generating pixels that look like falling. A robot cannot fake gravity. A simulator can approximate friction with a coefficient. A robot that applies the wrong amount of force drops the cup or shatters it. The consequences of prediction error in robotics are not visual artifacts. They are broken objects, failed tasks, and damaged hardware.
The result is a severe selection pressure. Only world models that genuinely capture the physics of interaction survive contact with reality. This makes the Roboticist’s Road slower than the others, less glamorous, less amenable to breathtaking demos. But it also makes it the most honest test of whether a world model actually works.
Prediction Conditioned on Action
What fundamentally separates the Roboticist’s Road from the others is not data scarcity, or contact dynamics, or the VLA debate. It is a conceptual distinction that the other roads can defer but this one cannot.
A video model predicts what happens next. A robot’s world model must predict what happens next if I do this.
The difference seems small. It is not. Unconditional prediction, forecasting the next frame of a video, can succeed by learning the statistics of natural scenes. The sun continues to set. The ball continues to fall. The pedestrian continues to walk. But action-conditioned prediction requires modeling the consequences of intervention. What happens to the cup if I push it? What happens to the fabric if I pull here instead of there? What happens to the tower of blocks if I remove this one?
This is the distinction between a movie and a model. A movie shows you what the world does. A model lets you ask what the world would do under different actions. Planning, the ability to evaluate multiple possible actions and choose the best one, requires action-conditioned prediction. And planning is what makes a robot useful.
Hafner’s Dreamer, back on the Dreamer’s Road, understood this from the start: the whole point of learning a world model was to plan inside it. But Dreamer operated in simplified environments, game worlds and simulated tasks, where the dynamics were clean and the state spaces were manageable. The Roboticist’s Road demands action-conditioned prediction in the full complexity of the real world: cluttered kitchens, novel objects, deformable materials, partial occlusion, and forces you can feel but not see.
This is why, despite Fan’s critique, the VLA approach has traction. In the absence of a world model that can predict the consequences of actions in rich physical environments, the field has settled for an approximation: train a policy that maps observations directly to actions, borrowing semantic knowledge from language models to fill the gaps. The world model is implicit in the policy, baked into its weights, never explicitly queried or planned against. It works, partially, for the tasks and environments covered by the training data. Whether it will scale to the open world remains the central question.
The Road Ahead
The Roboticist’s Road has achieved things that seemed impossible five years ago. A generalist policy can fold laundry, clean an unfamiliar kitchen, and generalize across robot platforms. But the field’s honest practitioners acknowledge how far the road still stretches.
Success rates hover between 20 and 60 percent on novel tasks. Every robotics demo is cherry-picked from dozens of attempts. There are no agreed-upon benchmarks, no standard evaluation protocols. Hardware reliability limits software iteration speed: a robot that breaks its gripper every few hours cannot collect the data needed to improve. The data problem has not been solved, only managed.
There are, however, early signs that the equation may be shifting. V-JEPA 2, discussed in Part 5, pretrained on over a million hours of internet video and then added action conditioning using just 62 hours of unlabeled robot data. The result was zero-shot robot planning in labs the model had never seen. Sixty-two hours is still physical data that must be collected, but it is orders of magnitude less than the 130,000 demonstrations RT-1 required for its single kitchen. If internet video can supply the bulk of physical understanding and robot data needs only to bridge the last mile to action, the defining constraint of the Roboticist’s Road may prove less absolute than it appears.
And the deepest question remains open. The VLA approach transfers semantic knowledge effectively: a robot that knows what a kitchen is can navigate one. But semantic knowledge is not physical knowledge. Knowing that cups are fragile does not tell the robot how much force will break one. The Roboticist’s Road needs world models that combine the semantic understanding of language models with the physical understanding that, so far, only comes from interaction.
The next part of the series steps back from the individual roads to ask the question they all converge on. If we need a world model that understands physics, responds to actions, predicts at multiple time horizons, and generalizes across environments, what should that model actually look like? Should it predict pixels, or something more abstract? Should it be a single monolithic network, or a modular architecture? Should it learn everything from data, or encode known physics as structure?
These are not engineering details. They are the architecture of understanding itself.
Next: Part 5, “The Architect’s Road,” examines the deepest question in world modeling: should a universal world model predict pixels or predict in a more abstract space? This is where Plato’s cave resurfaces as a live technical argument.
This is why I am interested in the possibility of using the robot as avatar with a human controlling it while the robot gains knowledge of the world and acting in it. It would seem like a faster way to "learn" and acquire data. Also porting one learned model into a new body has limits. The new body must be almost identical. If you were to wake up in another body you would stumble out of the gate and take maybe a short time or maybe a very long time to adjust. Were you a short person before and now you are tall? Robots will by necessity tend to be exact clones of one another to simplify this process. But more importantly many of the robots functions need to be imprinted at the limb level so that a controller mind can be dropped in only needing to know a basic set of commands like move legs, grasp with hand, etc. each joint will need local level processing and be coordinated with all of the other joints with the robot body basically capable of a limited level of autonomy. When an animal like a human gets its head cut off it is still capable of limited movement. A very creepy thing to witness but underlines the fact that bodies, human or animal, are actually two distinct "minds".
It is this two minds theory that raises more problems. What happens when one mind is built on one substrate and the controller mind another? How do they integrate and sync? What if the limb level controller acquires its own quirks and loses sync with the higher level mind? This happens in humans with diseases like Parkinson's. Human's for the most part have pretty robust error correction built in. But eventually it breaks down.
On another level this is emblematic of the whole idea that AI will soon become conscious or replace humans with super general intelligence. Consciousness and the ability to perceive and understand the world we as humans inhabit is built around more than language and more than just looking at pictures of putting one foot in front of another. It is millions if not billions of years of fine tuning systems within systems that got us here. Engineering can map out the territory of what they can see, but under it is a very complex terrain of feedback and synchronization that is very hard if not impossible to reduce to binary computation. Human engineers may solve the superficial problem of getting a robot to run, but that's just a flat two dimensional representation of what a human body goes through. The same with consciousness, you might write it out on paper but to be conscious and navigate the real world do not mistake the map for the actual terrain.
Couldn't resist a Borges reference.