Robotics' End Game: Nvidia's Jim Fan

**Sequoia Host:** And up first, I'm delighted to introduce my friend Jim Fan. Uh Jim leads the embodied autonomous research uh group at Nvidia, otherwise known as Nvidia robotics. Um I think that robot robots are just one of the most thrilling things that's going to happen. Uh a car basically is a big robot, but I'm excited for robots can go beep boop and lift things for us. And so, Jim was Jim was a standout at last year's I I sent, and we're delighted to have you back.

*[applause]*

**Jim Fan:** Thanks [applause] everyone. Thanks.

So, it was a summer day in 2016. Actually, right in this office that we're sitting,
*[snorts]*
there's a guy in shiny leather jacket, you know, big biceps, hurling this large metal tray. And on this large piece of metal, he wrote, "To Elon and the Open AI team, to the future of computing and humanity, I present you the world's first DGX-1." So, that was the first time I met Jensen. And as any good intern would do, I rushed to getting line to sign my name on it. So, can you spot it, my name? It's here. And can you spot another? That's Andre, right there. So, Andre, we're going to the Computer History Museum. I feel like a dinosaur. You know, back then, I had no clue what I was signing up for. And then, no one can describe what happened next better than Ilya himself. If you believe in deep learning, deep learning will believe in you. And oh boy, did deep learning believe in all of us big time.

Three step functions, 6 years. That's how all it took to bring us here today. The first tick, GPT-3, pre-training. Next token prediction is really about learning the rules of grammar, the shape of language. It's about simulating how thoughts and code and strings in general should unfold. 2022, InstructGPT, supervised fine-tuning, aligning the simulation to do useful work. 01, reasoning, using reinforcement learning to surpass imitation learning, and finally, auto research, accelerating the whole loop beyond what's humanly possible.

So, as Andre said, all the labs are getting to the final boss fight. So, for LLMs, they're in the thick of the end game. And honestly, I'm very jealous. Look at how happy Andre was, big smile on his on his face. The LLM folks are having the party of their lifetime. They're speedrunning AGI on mystical creatures literally called methos. So, why can't robotics get a piece of fun?

So, as any self-respecting scientist would do, I copy homework and I give it a new name. I call it the great parallel. So, instead of simulating strings, can we simulate next physical world state? And then we can align through action fine-tuning onto a thin slice of that simulation that matters for real robots. And we let reinforcement learning carry the last mile. And that's it. The great parallel, copying the LLM success. If you can't beat them, join them. So, please join me in a new episode, robotics, the end game. I'm sorry, I just couldn't resist. Nano bananas too good. Thanks, Demis.

So, how do we play the end game? It boils down to two things, model strategy and data strategy. Let's look at the model first.

The last 3 years were dominated by VOAs or vision language action models, and models like Pi and Groot fall in this category. So, we assume that the pre-training is done by a VOA, and we simply graph an action head on top of it. But really, if you think about these models, they're LVAs because the most amount of parameters are dedicated to language. So, language is first first-class citizen, followed by vision and action. And by design, VOAs are great at encoding knowledge and nouns, but not so much at physics and verbs. It's kind of head heavy in the wrong places. This is my favorite example from the original VOA paper. Move the Coke can to a picture of Taylor Swift. Yes, it has not seen Taylor Swift before. Yes, it's able to generalize, but this is not quite the pre-training ability that we're looking for.

So, what's the second pre-training paradigm? And I always thought that it would be something glorious. Unfortunate, it turns out that this is AI video slop that we call. You know, I can watch these um cats playing banjo on security cam all day. It's peak internet. But really, look at this. No one can take this seriously
*[laughter]*
until we realize that these video models are learning to simulate next world state internally. So, these are some rollouts from VEO-3. You can see that the models, they pick up gravity, buoyancy, lighting, reflection, refraction, all by themselves. None of this is coded in. Physics emerge by predicting the next blob of pixels at scale. And even visual planning emerges. Look at how VEO solves these mazes. It solves them by running simulation forward in pixel space. And draw attention to the lower right corner here. This is my favorite example. Let's watch, and you blink if you miss how VEO-3 solves this one. It's super smart. You know, VEO-3 figures out that if you're not looking, geometry is optional. I call this physics slop.

So, how do we make these world models useful? Well, we do action fine-tuning. We align the superposition of all possible future states, and [snorts] collapse that onto a thin slice that matters for real robots. Introducing Dream Zero. It's a new type of policy model that dreams a couple seconds into the future and acts accordingly. And you know that motor actions, they're high-dimensional continuous signals. So, that looks just like pixels. We can render it at the same time as we render the videos. So, Dream Zero Zero jointly decodes the next world states and next actions.

And as a result, it's able to zero-shot solve tasks and verbs that it has never seen in training. And as the robot executes, we can visualize what it's dreaming about. And the correlation is very tight. If the video prediction works, the action works. If the video hallucinates, the action fails. So, once again, vision and action are now first-class citizens.

And we have a lot of fun with Dream Zero. So, we just roll the robot around um in our lab, and then type random things into the prompt box. And of course, Dream Zero is not going to get all of these tasks 100% robust, but it's kind of like GPT-2. It's trying to get the shape of the motion correct in every case. So, Dream Zero is our first step towards open-ended open vocabulary prompting for robotics. And we [snorts] call this new type of model world action models or WAM. So, let's all take a moment of silence for our dear friend VOAs. They've served us well. Rest in peace. Long live world action models.
*[clears throat]*


And next, data strategy. This is Nvidia's chief scientist, Bill Dally, operating teleoperation inside our lab. And given his salary, I think this is by far the most expensive teleop trajectory ever collected in our data set.

The past 3 years have been dominated by teleoperation. It's the golden era. All right, VR headsets, extremely optimized latency for streaming, and these complex rigs that look like medieval torture devices. You know, so much investment in industry, so much pain and suffering. And yet, for teleop, it's upper bounded by 24 hours per robot per day, the fundamental physical limit. And actually, who am I kidding? It's more like 3 hours per robot per day, and only when the robot god is merciful because they throw all tantrums all the time.

So, how can we do better? Well, how about this? You just wear the robot hand on your own hand. So, this is called UMI or universal manipulation interface, and it's a deceptively simple idea. You wear the robot actuator on your hand and directly collect the data as humans, while getting the rest of the robot body out of the loop. Yet, I would say UMI is perhaps one of the greatest papers ever written in robotics data, and it spawned two unicorn startups. On the left hand side is Genesis, improving this design so you can wear the gripper here. And then on the right hand side, Sunday made these three-finger data gloves.

So, last year, we took it one step further. We designed this exoskeleton that has a one-to-one mapping with five-finger dexterous robot hands, and we call it Dex UMI. Let's look at it in action. On the left, the human directly collecting data always is fastest. On the right, look at how difficult teleop is. All right, the human operator, here one of our most skilled PhDs, he has to align very carefully, right? And then it's super slow. Also, the success rate is very low as well. And in the middle, you just exoskeleton and you collect data directly. And we train a robot policy on this data. So here what you see is a fully autonomous robot of a policy that's trained on zero teleoperation data. So we're able to break the curse of 24 hours per robot per day and see how happy these robots are because they no longer need to be in the loop for data collection.

So is this the answer? Have we solved scaling for robotics?

Anyone driving Tesla or Waymo here? Anyone? Right? You know, when you're driving, you're actually contributing to the biggest physical data flywheel. And the beauty is you don't even feel it during FSD because the data upload is an ambient process. Yet wearing these Umi or data wearables is still cumbersome, right? It's intrusive. It's not as seamless as just driving to work. So we need an FSD equivalent. The data collection needs to get out of the way, fade into the background so we can capture the full glory of human dexterity across all walks of lives, across all labors of economic value.

So we're going all in on human egocentric videos that come with these detailed annotations like hand position tracking and dense language annotations.

Introducing Ego-Scale. Where 99.9% of the training that goes into this is based on human egocentric videos. And the result is an end-to-end policy that maps directly from the camera pixels here to 22 degrees of freedom high dexterity robot hands. What you see here is fully autonomous.

We pretrain Ego-Scale on 21K hours of in-the-wild egocentric human data with zero robot data whatsoever. And during pretraining we predict these hand joints and wrist poses. Then action fine-tuning, we collect only 50 hours of high precision mocap data gloves and 4 hours of teleop. That's 4 hours of teleop. Less than 0.1% of our training mix. And with this Ego-Scale is able to generalize to these very dexterous tasks like sorting card or manipulating syringe. Right? Over transferring the liquid. You know, someday we might have robot nurses at home. Might as well try this. And for these tasks it takes only one shot demonstration at test time to learn different shirt folding strategies.

And perhaps the most fascinating finding from the paper is that we discovered this neural scaling law for dexterity. It's a very clean relationship between the amount of hours we put into pretraining and the optimal validation loss. In fact, it's a clean log-linear mathematical equation. Six years after the original neural scaling law for language models.

So if we put all of these data strategies on this chart, X axis is alignment to the robot hardware, Y axis is scalability, this is what it looks like. Teleop, the least scalable. Data wearables, you can go up to hundreds of thousands of hours. And egocentric video, if we're able to spin the FSD flywheel, easily 10 million hours in the next year or so. And if we draw a line here, everything to the left of this line is a new paradigm, sensorized human data.

So let me make a few predictions. In the next year or two, we'll see teleop dropping and dropping to almost negligible amount. And then there will be an ensemble of data wearables custom designed for different hardware and use cases. And finally, the main diet for robotics will be egocentric videos. So, a moment of silence for our dear friend teleop. You have served us well. Rest in peace. Long live sensorized human data.

Are we done with the data strategy yet? Did you notice I put two rings on data strategy? What's the outer ring here?

All the LM frontier labs have spent significant budget now on acquiring millions of coding environments to do reinforcement learning.

So robotics is the same. We're in urgent need to scale up environments. And of course, you can always do reinforcement learning directly on the real robot.

So in our lab, we use RL to push certain tasks to almost 100% success rate so you can do these continuous execution for hours on end. You know, it's kind of therapeutic to see these robots assembling GPUs just by themselves. Or as a wise man would say, good boy, this task has been approved by my boss.

Yet we can't get [snorts] to 1 million environments because that would require 1 million robots if you do it the previous way. So we need a better way.

Here, let's say you take an iPhone picture and you can pass this through this 3D world scan pipeline to extract all the objects and then automatically synthesize them again inside a classical physics simulator. So all these objects are actually interactive after the scan. And then you can augment this infinitely in simulation with variations that we call digital cousins. So now iPhone basically become a pocket world scanner in this process that we call real-to-sim-to-real. And in this way we have a scalable way to port the physical world into the digital world. But still this method relies on a classical graphics engine. Can we do better?

Introducing Dream Dojo. So it's our spin on video world model and turning them into full-fledged neural simulators. Dream Dojo takes as input these continuous action signals and outputs the next RGB frames as well as sensor states in real time. Not a single pixel you see here is real. And Dream Dojo is able to capture and learn the mechanics of different robots through a purely data-driven approach. There is no physics equation, no graphics engine involved in this process.

So the new post-training paradigm for robotics is a massively parallel RL system that runs on a few real robot stations, a bunch of graphics cores running world scans, and heavy inference compute running world models. Or as this equation goes, compute now equals environment now equals data. Or as a wise man would say, the more you buy, the more you save. And this message has been approved by my boss.

So that's it. Putting it together, the great parallel that robotics will follow. And it's happening as we speak. And we're looking at the beginning of the end game.

You guys play the video game Civilization? Still my favorite. I like to think of my research as unlocking game achievements on this civilizational technology tree. And there are three more achievements to unlock for robotics and then we're done. I can retire and I can't wait for that.

The first is passing the physical Turing test. Across a wide range of activities, you cannot tell the difference between a human doing a task or robot doing it. Maybe not drunk humans, but you know.

Physical Turing test is about unit energy in and unit labor out. And just by judging at the sexy pose of this robot, I think the work is cut out for us. So maybe it's two to three years away.

And next, physical API. You have a whole fleet of robots and they can be configured just like any other software using APIs and command lines, orchestrated someday by Opus 9.0.

And if we have this physical API, we'll be able to realize lights-out factories. Those are essentially printers of atoms. They take as input design in markdown files and then output fully assembled products, completely autonomous. Or these wet labs that automate scientific discoveries in chemistry biology and medicine.

And the final stop, physical auto research. When the robots start to design, improve, and build the next iteration of themselves far beyond what's humanly possible.

So you might ask, is this too science fiction? Like are we going to see this in our lifetime? it took the AI community 14 years to go from the first forward pass of AlexNet in 2012, a model that barely recognized cat versus dog, to AI ascent today, 2026, where we talk about agentic auto research.

And let's just add another 14 years. How about that? 2026 is right in the middle of 2012 and 2040. And technology does not advance linearly, it advances exponentially. So [snorts] I can say with 95% certainty that we'll get to the end of the end game, the end of the technology tree, by 2040. And we'll still be all We'll still be young. If you believe in robotics, robotics And to all of us here, sitting here, I think our generation was born too late to explore the Earth and too early to explore the stars, but we are born just in time to solve Thank you.
*[applause] [applause]*