Law Firm Says Test AI Agents Like Job Candidates Before Hiring

Before a company lets an AI agent place orders or move money, it should test the tool the way it would interview a new hire. That is the advice from Mayer Brown attorneys Spencer Glende and Samuel J. Hartman, in a Thursday (Oct. 22) article on Mondaq. AI agents are tools that can plan and carry out tasks with little human input.

Two traps to avoid

The lawyers say companies tend to fall into one of two traps. Some move too fast and launch agents before they know what the tools can and cannot do. That can mean wasted money, legal trouble or damage to a company's reputation. Others fear the risks so much that they skip tools that could deliver real value.

The article says agents "may bring game-changing gains or cause irreversible harm," depending on how they are used and controlled. Its answer is a well-designed pilot, which is a short, limited trial, backed by a strong contract. A pilot should ask three questions: Can it do the job? Is it safe? Will it pay off?

Write the job description first

The contract should spell out which tasks the agent will handle, what data and systems it can reach and who can work with it. It should also set limits. A purchasing agent, for example, might draft orders under a certain dollar amount, with a human approving them.

Test it hard

The article says companies should track several things during the trial:

• How often the agent finishes its tasks
• How accurate it is
• How often it makes things up or goes past its limits
• Its performance in tricky situations, confusing requests and fraud attempts

The agent should also keep detailed records of what it did and why. Safety checks need to go beyond rules on paper. A company should confirm the agent really respects spending caps and privacy rules, and should test how fast an emergency off switch works. Too many alerts can backfire, because busy staff may start ignoring them.

A contract is not enough

A contract can decide who pays when something goes wrong. It cannot stop the failure from happening. That is why hands-on testing and careful vetting of the vendor still matter.

Cost is the third test. Software fees are only part of the bill. Computing charges and the extra staff needed to handle problems can add up quickly. An agent might replace some workers but require more supervisors.

What happens next

Every pilot needs a clear end date and a firm decision point, the article says. A test should never slide into full use just because no one stopped it. The contract should also cover a shortfall. Options might include a longer trial, more fixes or a smaller rollout, instead of a simple yes or no.