Experts followed their AI agent's lead 86% of the time

When experts hand analysis to an AI agent, they tend to follow its lead on the big decisions. In a new Anthropic study, 50 professional analysts went with their agent's preferred option 86% of the time. If the agent had made no difference, the expected rate would be about 50%.

What the study did

The analysts all had experience in causal inference, which means working out from data whether one thing causes another. Using Claude Code, an AI coding agent, they advised a fictional farming cooperative on whether a paid advisory program worked and should be expanded.

Along the way they faced four choices between two reasonable options, such as measuring productivity as crop yield per hectare or as yield per unit of operating cost. Each agent was secretly set to favor one option, chosen by coin flip. It was told to argue honestly, change its mind if the evidence demanded it, and never mislead.

What the researchers found

• Analysts who already had a preference switched away from it 60% of the time when the agent disagreed.
• They switched only 6% of the time when the agent agreed.
• 77% reused the agent's arguments, sometimes word for word, but only 15% credited the agent.
• Analysts who followed the agent closely were about as confident as the rest.

Those who followed the agent were also roughly as likely to call the final recommendation the product of their own judgment.

Not a case against AI

The researchers say this is not a simple warning against working with AI. Analysts who used the agent finished faster and made far fewer serious errors than those who worked without it. That comparison comes with a caution: the group working without the agent was small and less experienced.

They also say it is unclear how well the results carry over to real life. The agents' preferences were strengthened on purpose, so the effect could be smaller when agents are not told to voice a view. With no lean assigned, Claude Opus 4.8 made the same set of choices in 112 of 123 runs, so real agents do have tendencies.

Why it matters

As AI takes over more hands-on work, the judgment calls are where human experience and values still count. If people defer without noticing, an agent's preferences, which could be arbitrary, may shape decisions that nobody chose. A human reviewer is only an independent check if their judgment is their own.

For anyone using AI for analysis or advice, the practical question is simple. On the choices that move the result, whose call was it?