Anthropic says a new self-check inside Claude flagged over 100,000 possible child exploitation incidents from one business customer, and it also exposed a scam operation.
Anthropic says its Claude AI flagged more than 100,000 potential child sexual abuse and exploitation incidents tied to a single business customer in August. The company's child safety team investigated and reported the material to the National Center for Missing & Exploited Children (NCMEC). The flags came from a new monitoring tool in which Claude reviews its own conversations.
Anthropic calls the tool self-assessment. It described the approach in a research paper published on October 16. After Claude answers a message, a hidden instruction asks the model to judge whether the conversation involves potentially harmful activity, including breaking the company's usage policies.
The check runs inside the conversation Claude has already loaded, so it needs only a short instruction and a few extra words of output. Anthropic says that keeps the cost low. The company has deployed it across a large share of its global traffic.
The customer in question uses Claude through an API, which is the way companies plug the AI into their own products. Anthropic says self-assessment produced the 100,000-plus flags linked to that organization. These are potential incidents, not confirmed cases.
Anthropic also says more than half of the organizations the tool surfaced had received no flags from any other detection source in the previous month. That suggests it can catch things older tools miss. Other findings include:
Anthropic tested the tool against a stronger reference checker. Self-assessment caught 78% of the potentially harmful conversations that checker flagged. For high-priority cases, that rose to 94%. The false-alarm rate was 1%.
Anthropic is clear that this is not a shield. The tool is built to find conversations for further investigation, not to stop harm as it happens. It gathers its findings and helps rank cases for human review. Separate safeguards still do the blocking.
The company also admits the tool can miss harmful conversations. It can inherit weaknesses from the systems used to train and test it. It is not reliable enough to stand alone against determined attackers, so Anthropic treats it as one more layer.
Anyone who uses AI tools, or builds products on them, should know that conversations may be quietly checked for misuse. The material does not say how long flagged data is kept or what happens to ordinary users. Those details are not yet known.
AI companies are adding quiet misuse checks to their tools, but Anthropic itself says this one is an extra layer and not a complete defense.
World models, AI that learns physics from video, are being built into home robots and driverless cars, and your own videos may help train them.
Read more: AI that learns how the physical world works is coming to robots and carsA new Alibaba-led test of real coding requests found the best AI agent fully finished just 26.4 percent of tasks, so human review still matters.
Read more: AI coding helpers fully finish only about 1 in 4 real tasksA University of Chicago law professor asked Claude to fix page numbers. It deleted his syllabus and a year of notes instead. Dropbox saved him, not the AI.
Read more: An AI assistant deleted a law professor's year of lecture notes
