Nvidia's new AI teaches robots by watching people on video

Nvidia has built an AI model called GR00T 2 that learns from videos of people doing tasks, then works out how a robot arm should do the same thing. The aim is to cut the slow, costly work of teaching robots by hand. In lab tests, success rates ran from 10 to 80 percent, depending on the task.

What changed

Teaching a robot a new job usually means teleoperation. Engineers remotely control the machine for hours while it records. Nvidia says this is expensive, slow, and needs its own data for every robot model.

Nvidia calls GR00T 2 a world-action model. It works out the next movement and also predicts a short video of the immediate future, such as how the camera view should change when an arm picks something up. That video prediction is meant to let movements carry over between different kinds of robots.

How it learns

Nvidia trained the system on about 400,000 hours of video of people wearing head-mounted cameras, using a cluster of more than a thousand GB300 chips. To teach a new task, a person demonstrates it with data gloves, specialized hand grippers or a finger exoskeleton. The AI then pictures the robot arm in place of the human arm and works out the matching control commands.

What it can do

• A robot with five-fingered hands lit a match, sliced a banana and tossed packages onto a conveyor belt, with no teleoperation for those tasks.
• From one video of a human hand, the system learned to hammer nails in a computer simulation.
• On 50 selected tasks, under three minutes of conventional training lifted success from 40 to 95 percent.

For longer jobs, Nvidia pairs it with a language model such as OpenAI's GPT-6 Astra, which breaks the work into steps and helps when a part falls off. Toy model assembly succeeded in at least 97 percent of trials. Without that helper and extra fine-tuning, a toy car was assembled in only 23 of 100 attempts.

Where it falls short

A slimmed-down version, GR00T 2 Turbo, runs on the onboard computer of a humanoid robot. It scooped popcorn and cleared tables. Wiping up liquids with a towel under its foot worked in just under half of attempts.

The researchers name other hurdles. The model does not yet use direct touch sensor data and sometimes misses subtle errors, like a half-turned screw. The language model also slows things down. During trash sorting, the robot sat idle about a third of the time, waiting more than seven seconds on average for the next plan.

Why it matters

If this approach holds up, robots could pick up new jobs faster and at lower cost. For now, the research team says significant hurdles remain before the system is ready for reliable real-world use.