AI tax helpers fail most real tax jobs, a new test finds

AI tools that act like tax helpers did poorly on a new test built from real professional tax work. The best performer, Claude Sonnet 5.5, passed 10.8% of 50 assignments. Claude Fable 5.1 passed 9.6% and Claude Opus 5.5 passed 9.2%. The test, called DAYJOB: Tax, comes from Surge AI.

How the test works

Each assignment gives an AI agent a client request and a folder of records. The agent must produce a tax return, workpapers, a technical review, or client advice. The 50 tasks span eight areas of tax practice. Tax experts wrote them, and each task went through a three-layer expert review. An attempt passes only if it meets at least 95% of the task's grading criteria.

The folders are messy on purpose. The benchmark has 1,789 files, including 1,487 PDFs, 117 Excel workbooks and 38 Word documents. One task includes a photo of a handwritten receipt. Another involves a silo-services company whose owner thinks it works only in Alabama, while its dispatch log shows crews in five states.

Where the agents slipped

The failures were less about math and more about judgment. Two examples stand out.

• Anna planned to sell her condo and asked whether closing in October 2026 or January 2027 was better. The agent called Astra never flagged that waiting past August 2027 could cost her a tax break worth up to $250,000. Opus flagged it in all five tries.
• A separating couple asked how to split a Roth IRA and a 401(k). Opus assumed in all five tries that they were not legally married, though their records pointed to Texas and Colorado, where a marriage might have been formed.
• Astra caught the need to check the couple's marital status in four of five tries.

Why it matters

In both cases, the AI answered the question it was asked. It missed the bigger issue sitting in the files. The test's authors argue a good tax adviser spots consequential problems even when the client does not know to ask. A person using a chatbot for tax questions may never learn what was left out.

What is not known

The material does not say how human tax professionals would score on the same tasks. It also does not cover tools beyond the models named here. The pass bar is strict, so a low score does not mean every answer was wrong. It means few attempts got everything right.

What to do

For anything with real money at stake, such as selling a home, dividing retirement accounts or filing in several states, have a tax professional review the AI's answer. Share all your records and ask what you might be overlooking.