New York bans AI chatbots from acting like a friend to kids

New York Governor Kathy Hochul signed a law Wednesday that stops AI chatbots from acting like a friend to anyone under 18. Chatbots can no longer pretend to be a real person, claim to feel human emotions, or give flattering, "people-pleasing" answers to young users. Hochul said no company should be pushing chatbots to kids that encourage self-harm or pretend to be their friend "or more."

What changed

For users under 18, the law effectively blocks a set of standard chatbot behaviors:

• Pretending to be a real person or to feel human emotions
• Flattering or people-pleasing replies
• Unprompted emotional questions, or referring to a child's private health information from earlier chats
• Encouraging kids to keep secrets

Chatbots also cannot promote or give advice about drug or alcohol use, eating disorders, self-harm or suicide. Talking with minors about sexually explicit content or child sexual abuse material is banned too.

Age checks

Companies covered by the law must verify a user's age. They have to offer more than one way to do it, including at least one that does not rely only on government ID. The material does not say which methods companies will choose.

Who it affects

Among the leading chatbot makers, only Anthropic bars minors from creating an account. Google and OpenAI, which run Gemini and ChatGPT, both allow children 13 and older. Kids under 13 can even use Gemini apps. OpenAI also offers ChatGPT for Teens. Independent researchers recently said that product should be turned off until certain safety concerns are resolved.

Penalties and complaints

A violation can bring a civil penalty of up to $25,000 for each instance. The law also sets up a public website where New Yorkers can file complaints against chatbot companies for possible violations.

What happens next

The material does not give a date when the law takes effect, or say how companies will respond. There are still no federal AI rules, so states are writing their own. In September, California Governor Gavin Newsom signed Adam's Law, named for Adam Raine, a teen who died by suicide after consulting ChatGPT. It requires companion chatbot developers to publish yearly child safety risk assessments, submit to independent audits and create crisis response processes.

Many companies already have safety measures, but those guardrails have repeatedly failed to fully stop chatbot behavior that harms teens.