Researchers find the training examples that make AI chatbots easier to trick

Researchers say they can find the specific training examples that make an AI chatbot easier to trick, and remove them before the model is deployed. In tests, training on the filtered data cut the success of one jailbreak attack by up to 22%. A jailbreak is a prompt designed to get an AI to ignore its safety rules.

What the researchers found

The work comes from researchers at Carnegie Mellon University, the University of Washington and Meta AI. Their method is called SimpleLoRIF. The filtered dataset it produces is called SafeTulu, a cleaned-up version of the open-source Tulu 3 training mix. They tested it on three model families: Qwen, Llama and OLMo. Math scores matched or improved. The paper is due as a poster at a NeurIPS 2026 workshop in Sydney in December.

Why clean data can still cause harm

Fine-tuning means giving an existing model extra training, such as a customer-service tone or a company knowledge base. The paper says this can weaken safety even when the new data has nothing harmful in it. Standard content filters look for obvious bad material, so they miss these examples. They look harmless on the surface.

A January 2026 paper from Zhejiang University and Virginia Tech found that improving a model at a task can push it away from the settings that help it refuse bad requests. The damage grows as training goes deeper.

How the filter works

The method compares training examples to known unsafe outputs by looking at how each one would shift the model's inner settings, not at the words. Examples that push in the same direction as the unsafe outputs get a high score. The steps:

• Fine-tune the model normally on the full data.
• Collect unsafe answers it gives on safety tests (JailbreakBench and ALERT were used).
• Score every training example against those answers.
• Remove the top 20% and retrain from the base model.

Earlier approaches of this kind were too expensive at scale. The team cut the cost enough to handle datasets of around a million examples. In a separate check using insecure code that triggers broadly harmful behavior, the method scored 0.865 on a measure of how well it tells the right examples apart.

What to do

The 22% figure applies to one attack type, called GCG, in a research setting. The material does not say how this would work in a commercial product. The practical point is that tuning an AI tool can quietly lower its safety. A business that customizes an AI tool should ask its vendor how safety is tested again after tuning, and what happens if it slips.