Table of Contents
TL;DR: Six real failure modes, six different people, six different industries. The marketer who shipped an AI-generated product description that promised features the product doesn’t have. The attorney who cited a case that doesn’t exist. The novelist whose grief scene reads like a sympathy card. The consultant whose 80-page report turned out to mean something different than what they thought. The doctor’s office chatbot that traps the patient for five minutes. The founder whose business plan sounded right and was wrong in three specific places. Read all six and the pattern becomes obvious. Then go look at your own AI workflow.
1. The marketer
She works for a mid-sized consumer electronics brand. Product launch in ten days. The copy team is behind. She asks AI to draft product descriptions for the new line of wireless earbuds, fast.
The descriptions come back clean. Tight benefit copy, the right brand voice, every spec listed. She skims them, runs spellcheck, sends them to the e-commerce team for the site upload.
Two weeks later customer service is on fire. The AI confidently described a noise-cancellation feature the earbuds don’t have. Hundreds of customers bought the product expecting that feature. The company has to issue refunds, rewrite the copy, and explain to leadership how the wrong feature ended up on the product page in the first place.
The lesson: AI invents specifications when the prompt asks for them and the prompt didn’t constrain what was actually true. The fix is to feed the model the verified spec sheet, not ask it to remember what your product does. Confident invention is the failure mode, and product copy is exactly the wrong place to find it.
2. The attorney
He’s a senior partner at a law firm in New York. Brief due Monday, late Friday afternoon, his associate has dumped a draft in his inbox. He skims it. The argument is solid. The case citations look good. He files it.
Two weeks later he’s in the judge’s chambers explaining why three of the cases cited in his brief do not exist. The associate used AI to find supporting precedent. The AI invented citations that looked exactly like real citations, complete with plaintiff, defendant, court, year, and page numbers. The associate didn’t check them in Westlaw because they sounded right. The partner didn’t check them because the associate was supposed to.
The judge issues sanctions. The story makes the legal trade press. The firm spends six months in damage control.
The lesson: AI invents authoritative-looking citations as easily as it generates the rest of the sentence. There’s no signal that the citation is fabricated. The only fix is verifying every citation in an independent database before filing. I get into the full mechanism of this failure mode in AI Hallucination: A Survival Guide for People Who Publish Under Their Own Name.
3. The novelist
She’s writing her second book. Memoir-inflected fiction, the kind of story where the emotional moments have to land hard or the whole project dies on the page. Chapter twelve is the funeral scene. She’s been avoiding it for three weeks because she can’t get it right. So she asks the AI to draft it.
The draft comes back competent. The grammar is fine. The structure is fine. The grieving daughter cries the right number of tears in the right paragraphs. Her best friend says the right comforting things. The chapter ends on the right note of acceptance.
And it reads exactly like a Hallmark sympathy card.
The novelist reads it twice trying to find what’s missing. What’s missing is the specific. The one detail only someone who had actually buried someone they loved would know to include. The smell of the funeral home carpet. The wrong song the music director picked. The relative who showed up drunk. The pasta salad someone brought in a tupperware container that didn’t have a lid. The machine has no memory of grief. It has read about grief. It writes the average of everything ever written about grief, and the average is hollow.
The lesson: AI cannot write from inside an experience. It can only write about one from the outside. For anything where the reader needs to feel something real, the writing has to come from a human who has felt it. I had a client send me his ChatGPT 3.5 drafts of a memoir he was proudly working on. He kept sending me more, trying to help. I ignored them and did interviews to get what he actually lived through. He genuinely liked the AI versions better. That cost real back-and-forth time to resolve.
4. The consultant
He’s billing $40,000 to a Fortune 500 client for a strategic analysis. The deliverable is an 80-page report with recommendations. Tight deadline, he asks the AI to help him synthesize his research and draft the body.
The report comes together in three days instead of three weeks. He reviews it, edits the executive summary, adds his own framing in the introduction and conclusion, and ships it.
Three months later the client is implementing one of the recommendations and discovers that the supporting analysis in the report doesn’t actually support the recommendation. The data tables are correct. The conclusion drawn from them is subtly wrong. Somewhere in the middle of the 80 pages, the AI’s reasoning drifted in a way that was hard to spot if you were reading section by section but obvious if you mapped the argument end to end.
The client doesn’t fire him. They just don’t hire him again.
The lesson: AI drifts across long documents. The argument bends slowly with each new section, and twenty pages later the document is making a different argument than the one it started with. The fix is human review of the through-line, not section-by-section editing. I cover this specific failure mode in AI Drift: The Failure Mode Nobody Talks About.
5. The doctor’s office
The medical practice deploys a new AI chatbot to handle incoming patient calls. The system can answer questions about office hours, scheduling, prescription refills, and routine billing. On paper, this should free up the front-desk staff for the calls that need a human.
In practice, the system answers every call with a two-minute uninterruptable speech about how it’s going to help, then a one-minute follow-up about what it can do, then more after that. Five minutes every call, every time, before a patient can leave a message or reach a human. The chatbot has no off-ramp for the patient who already knows what they want and just needs to leave a message.
Patients complain. Some switch to a different practice. The system technically works as designed. The design didn’t account for the case where the user just wanted to talk to a person.
The lesson: AI deployed without an exit path for the user becomes a trap. Every customer-facing AI needs an obvious, fast, no-friction way to reach a human. The clinic didn’t think about the edge case where the user wanted to skip the AI entirely. So every patient calling in got the long-form AI experience whether it served them or not. The Klarna playbook at small scale, with the same lesson at the end.
6. The founder
She’s pitching a Series A. The deck is solid, the traction numbers are real, the team is strong. She asks the AI to help her draft the long-form business plan to attach to the deck. Market sizing, competitive analysis, financial projections, the whole package.
The plan comes back clean. She reads it, makes minor edits, sends it to the lead investor.
The investor’s analyst catches three problems in the first hour. The total addressable market figure is roughly double what the actual industry data shows. One of the listed competitors stopped operating eighteen months ago. A revenue projection assumes a customer acquisition cost that nobody in the category has ever actually hit.
The deal doesn’t die immediately. It gets harder. The investor now has reason to question every number in the deck. The founder spends the next two weeks re-doing analysis that should have been right the first time, and the term sheet that comes back has a lower valuation than the one she would have gotten on the original numbers.
The lesson: AI is shallow. It produces work that looks right because it sounds like what work in this category usually sounds like. The specifics are where the failure lives, and the specifics are exactly what a sophisticated reader checks. The Death of Thinking covers the broader version of this failure mode, what happens to a profession or a culture when shallowness becomes the default.
The pattern
Six different people, six different industries, six different failure modes. Confident invention. Hallucinated authority. Hollow emotion. Slow drift. No exit path. Shallow specifics. Read them next to each other and the pattern is obvious.
None of these failures were caused by bad AI. The AI did exactly what AI does. The failures were caused by humans handing the AI a job it wasn’t built for and trusting the output without verification.
The fix is the same in every case. Put the AI on the parts of the work that match its actual strengths, the parts covered in What AI Is Actually Good At. Keep humans in the loop everywhere else. Map the edge cases before you deploy, not after a customer or a regulator finds them for you.
That’s the whole job, and it’s the job most companies aren’t doing right now.
Frequently Asked Questions