Google tests an AI helper that watches you work and speaks up only when needed

A research team led by Google has built StreamVibe, an AI helper that watches you cook or assemble furniture and speaks up only when you seem to need help. The early results are mixed. The tests show how hard it is for AI to know when to talk, especially as a task drags on.

What it is trying to do

Researchers from Google, Stanford, Oxford and several U.S. universities worked on the project. Most video assistants today follow clear commands, such as "Let me know when the water starts boiling." StreamVibe is meant to handle vaguer ones, such as "Please help me if I run into any difficulties."

The paper says existing assistants also lose track of what has already been done. That leads them to repeat advice or answer too late. When a user changes a request midway, they often stick to the original one.

How it was tested

The team built a benchmark, a standard test, with about 4,100 examples drawn from more than 183 hours of video. The footage covers everyday life and software use. Four human reviewers had to agree on every example. The test has three stages: clear instructions, vague goals, and requests that change over time.

The researchers then trained their own model, StreamVibe-VL. It appears to be based on Alibaba's Qwen3.5-9B. Every second, it decides whether to stay quiet or speak. A second training step makes it more reserved so it does not interrupt constantly. A separate memory system tracks open requests and finished steps.

Where it falls short

The researchers report that StreamVibe-VL improves timing and quality of responses by 10.7 percent over other models. The memory system adds another 12.3 percent for proactive help. Google's own Gemini 3.8 Flash still scored higher in several tests.

The bigger finding is about longer tasks. When models had to build on their own earlier answers, errors piled up. One key measure of answer quality fell by up to 61.5 percent. In videos longer than five minutes, the tested models rarely picked the right moment to speak.

• In a photo-editing tutorial, one rival model said the image was saved before the user had opened it.
• StreamVibe-VL instead gave the next steps at the right time, such as "Colors" and then "Desaturate."
• In a toast-and-jam recipe, other models only described the screen. StreamVibe-VL advised cutting off the crust first.

What is not known

The researchers say real life is likely more complex than their test setup. The material does not say whether or when this will become a product people can use. For now, it is research that shows where AI helpers still stumble.