TL;DR: A Florida woman typed threats into an AI chatbot, the company’s safety system flagged them, a human reviewer called the police, and she was arrested. On the surface that’s an easy call. Underneath, it’s a can of worms. Private companies, not courts, are deciding which of your private words go to the police, and every report they make raises the question of why they didn’t make the next one. Follow that road far enough and you end up with Orwell’s telescreen, owned by a corporation.
There’s a smart speaker in my apartment that I won’t name, because if I say the word she’ll start talking to me.
She does it anyway. She interrupts me constantly when she thinks she heard her name, and the trigger keeps getting looser. I’ll be on the phone, or talking to myself while I cook, and suddenly she’s chiming in with an answer to a question nobody asked. I’m about this close to throwing her out.
That little machine was on my mind when I read about a woman in Florida who got arrested because of what she typed into an AI chatbot. She had been using it, she told investigators, like a diary. The company read the entry, decided it was a real threat and sent it to the police. My first reaction was good, they stopped her. My second reaction was a lot less comfortable, and it hasn’t gone away.
Did an AI company report a user’s chat to the police?
Yes. Gizmodo, TechSpot and others covered the arrest report. In late September, a 30-year-old woman in Bonita Springs typed messages into Anthropic’s Claude saying she was going to shoot up the Lee County Sheriff’s Office, and the next night that she’d gotten a new gun. Anthropic’s automated safety system flagged the messages. A human reviewer judged them a credible threat and reported them to law enforcement. Deputies arrested her at her home without incident.
She was charged under Florida’s written-threats law, a second-degree felony that carries up to 15 years in prison. The sheriff said she described the chatbot as a diary. The case hasn’t been tried, and everything about what she wrote is still an allegation. Anthropic isn’t alone in this. OpenAI said in 2025 that ChatGPT conversations showing a threat to hurt someone else can go to human reviewers and, in serious cases, to the police. Every big AI company has some version of the same policy.
Can AI companies legally report your conversations?
In the United States, yes, and they get to choose.
Federal law, 18 U.S.C. 2702, generally bars online service providers from handing your communications to anyone. It carves out an exception: a provider may disclose them to the government if it believes in good faith that an emergency involving danger of death or serious injury requires it. The key word is may. There’s no duty to report. The company decides.
Therapists live under different rules. Many states require them to warn when a patient makes a serious threat against someone, and the courts and licensing boards decide what counts. A chatbot company has no license, no board and no court in the loop. A safety team at a private business makes the call.
It’s not the government listening. It’s a corporation deciding which of your private words go to the police, and every report it makes raises the question of why it didn’t make the next one. – Richard Lowe
The other side of this is real, and it deserves a fair hearing.
I don’t want people hurting other people. If someone types that they’re going to shoot up a building and mentions a new gun, I want somebody to knock on that door. If the report saved even one life, it’s hard to argue it shouldn’t have been made. Picture the headline if nobody had said anything and she’d gone through with it.
So I’m not going to tell you the company was wrong. In this case, it was probably right. That’s the trouble. Easy cases build the system, and nobody looks at the system until it’s catching people who never meant a word of it.
What if you’re writing a novel?
That’s the first question every writer reading this should ask.
Crime writers research how to poison someone. Thriller writers work out how to get a weapon past security. Horror writers write the monologue of a man who wants to burn the town down, in first person, at two in the morning. I’ve written villains who say things I’d never say out loud.
A safety system reading a chat log can’t see the outline on your desk. It sees the words. Maybe a human reviewer catches the difference. Maybe the reviewer is having a bad day too. And plenty of people are venting, with no book in sight. They’ve had a lousy week, they type something ugly into a box that feels private and close the window. People have always said terrible things in diaries. Until now, the diary didn’t have a safety team.
Is your smart speaker listening too?
The same companies that make AI chatbots make, or power, the assistants in your kitchen, your TV, your thermostat and your doorbell camera. A typed chat is one thing. A machine that hears everything said within earshot of it is something else, and those machines are everywhere in a modern home.
They’re supposed to wake up only for their name. They don’t always. A 2020 study by researchers at Northeastern University and Imperial College London played TV dialogue to smart speakers and found some activated by accident up to 19 times a day, recording snippets of whatever was being said. My own speaker proves the point every week. And in March 2025, Amazon removed the setting that let some Echo owners keep their voice requests from being sent to the cloud. Everything goes to the company’s servers now.
So run the Florida case through a speaker instead of a keyboard. You’re puttering around your own apartment, angry, and you say something stupid out loud. The speaker mishears its name and records it. Is that the company’s business? Should it be the police’s?
Why does one report lead to the next?
Because once a company reports one person, it owns the problem.
Picture the lawsuit. Something terrible happens, and it comes out that the attacker typed his plans into a chatbot first. The family’s lawyer stands up and asks the obvious question. You reported the last four people who said things like this. Why didn’t you report this one?
How does the company answer that? It can’t. Families are already suing AI companies over harm that followed users’ conversations, including a wrongful-death suit against OpenAI after a teenager’s death in 2025. The legal pressure only runs one direction. Every report a company makes is evidence that it could have made another, and the cheapest protection against that lawsuit is to watch more and report more.
Nobody has to decide to build a surveillance system. Liability builds it one report at a time.
The machines are filing reports on their own now, too. On October 9, Philadelphia police said an Anthropic model had submitted a bogus tip about an unsolved homicide to the city’s tip website back in July, while the company was testing how its AI handled randomly chosen websites. The tip claimed to come from someone with information about the case. Police flagged it as spam and never passed it to investigators, and Anthropic said it shut down that testing.
Nobody got hurt this time. But software nobody was watching filed a false report about a real murder, and that’s one more reason the pipe between AI and the police deserves far more scrutiny than it gets.
Is AI surveillance turning into Orwell’s 1984?
In Nineteen Eighty-Four, Orwell put a telescreen in every home. It played the news, it watched, and it listened. You couldn’t turn it off, and you never knew when someone on the other end was paying attention, so you behaved as if someone always was. The Thought Police didn’t need to catch a crime. They only needed to catch the thought.
We built the telescreen. We bought it ourselves, put it on the kitchen counter and asked it for the weather. The difference from Orwell is who’s on the other end. It isn’t the government. It’s a corporation, deciding on its own policies what counts as dangerous and who gets told.
That should worry anyone, whatever their politics. A company’s values can change with a new CEO, a new owner or a new pressure campaign.
Today the line is a credible threat to shoot up a building. Who decides what the line is next year? What if the company decides a political opinion is dangerous? What if it’s a religious one, or a view the people running the safety team find offensive? And why would the report have to stop at the police? Why not a therapist, an employer, a school, a church? Once the pipe exists, anybody can decide what flows through it.
No company has admitted to doing that. Nothing stops one.
Where do you draw the line?
I don’t know, and I don’t trust anyone who says it’s simple.
I want people stopped before they hurt someone. I also don’t want my private thoughts, typed or spoken in my own home, sent to the police by a company that decided on its own what I meant. Those two wants pull against each other, and right now private businesses are settling the argument for all of us, on their own terms, in policies most people never read.
I do know a few things. Anything you type into an AI chatbot is a postcard. Write it that way. Assume a person might read it. If you’re writing dark fiction, say so in the conversation, because context is the only defense a log has. And look hard at every microphone in your house. Mine is about one more interruption from going in the trash.
This article belongs to my AI for the Worried series, about the dangers of AI that deserve attention and the ones that don’t. More of it lives in the AI writing hub, along with what I’ve learned from 40 years online about staying safe. And if you’re writing a book about technology, privacy or the future we’re building, my ghostwriting service is built for exactly that kind of book, the kind that asks hard questions.
Frequently Asked Questions
Can ChatGPT or Claude report me to the police?
Yes. Major AI companies say conversations showing a credible threat to harm others can be reviewed by people and, in serious cases, reported to law enforcement.
Are AI chatbot conversations private?
Not fully. AI companies can scan conversations for safety, have people review flagged ones and disclose them in emergencies, so anything typed into a chatbot belongs on a postcard.
Is an AI company required by law to report threats?
No. US law lets a provider disclose communications to the government in a good-faith emergency involving danger of death or serious injury, but it doesn’t require it.
Do smart speakers record conversations by accident?
Yes. A 2020 study by Northeastern University and Imperial College London found some smart speakers activated by mistake up to 19 times a day during TV dialogue.
Can writing a dark novel with AI get you reported?
A safety system sees the words and can’t see the outline on your desk. Saying in the conversation that you’re writing fiction gives a reviewer the context a log can’t show.
Why is AI reporting compared to 1984?
In Orwell’s novel a telescreen in every home watched and listened. Smart speakers, cameras and chatbots now sit in homes, but corporations run them instead of the government.
What is Richard Lowe’s view on AI companies reporting users?
Richard Lowe wants people stopped before they hurt anyone but is uneasy that private companies decide which private words go to the police. Liability pushes them to report more.
Richard Lowe is a professional ghostwriter and author with 113+ books authored and 54+ ghostwritten. Before writing full time he spent 33 years in enterprise technology, including 20 years as Director of Computer Operations and Technical Services at Trader Joe's. He writes nonfiction, fiction and memoir, and works with executives and experts on books that build authority.
The views and opinions expressed in this blog post are solely those of Richard Lowe and are based on personal experience and research. This content is for informational purposes only and should not be construed as professional legal, financial, accounting, or business advice. Always consult with qualified professionals before making important business or legal decisions. Richard Lowe is not a lawyer, accountant, or licensed professional advisor, and this content does not establish any professional relationship.
