Researchers teach robots to ask what would have happened if they'd acted differently

A team led by researchers at Brown University and NYU has described a method that lets a robot learn from a surprise. After something goes wrong, the robot asks a simple question: what would have happened if I had acted differently? The work is a preprint paper, and the results so far come from simulated tasks.

What changed

Robots are often trained with world models. These are AI systems that predict what will happen after an action. The catch is that they predict an average outcome. They struggle to say what would have happened in one specific event that did not go as expected.

The paper is by Randall Balestriero, Kerda Varaku and Turing Award winner Yann LeCun. It is titled "Counterfactual World Models: Natura Non Facit Saltus." The Latin means "nature makes no leaps."

How it works

Balestriero gave a boat example. A model predicts where a boat lands after steering left. The boat actually ends up two meters further downstream, because of a hidden current. To guess where it would have landed after steering right, take the model's prediction for steering right and add the same two meters.

He says this works because over short time steps, the effects of the action and the hidden disturbance simply add up, like forces. He also says the approach is not tied to one kind of model. It can be applied to any world model that predicts the average next outcome.

What the numbers show

• In simulated tasks, robots reached the goal in 35 to 69 percent of surprise events.
• A planner that knew the true simulator but ignored the observed outcome succeeded 14 to 23 percent of the time.
• With two more attempts on the same event, success rose to 51 to 97 percent.

Where it could be used

Balestriero named three likely uses. One is retrying an action after a surprise, such as an unexpected load or a weak motor. Another is adapting to a lasting fault, like a miscalibrated motor. The third is helping people understand why a failure happened and tell a bad decision from bad luck.

He said the experiments used off-the-shelf image encoders, called DINOv2 and MAE, with no fine-tuning and only a small predictor trained on top. Building this kind of model from that starting point would need only minor additions, he said.

What is not known yet

The tests were in simulation. The material does not say how the method performs on real robots in warehouses, factories or homes. The paper is a preprint, so it has not been confirmed by outside review. The researchers also said their first attempt, which used a learned component to infer the hidden disturbance, was too fragile. That led them to the current approach.