The Cracks in AI Search Nobody’s Warning You About

In the first part of this series I argued that AI search doesn’t kill your website, it raises the stakes on having a good one, and I meant every word. The opportunity is real. I wouldn’t have spent months of my own time chasing it if it weren’t. Stopping there would be lying to you by omission. The same machines that create that opportunity are broken, and almost nobody in the business of selling AEO will tell you how.

I will. I went deep into this for my own business, and a lot of what I found was ugly. These are the cracks to know about before you build. Some are merely frustrating. Some do real damage to your name while you’re not looking.

In old-fashioned search, you could at least see the board. There were tools that told you where you ranked for a given term, whether you were moving up or down, what was working. It was imperfect, but it was legible. You could form a theory, make a change, and watch what happened.

AI search gives you almost none of that.

There’s no reliable rank tracker for whether ChatGPT recommends you. The criteria these engines use to choose a source are opaque, and they’re opaque even to the people who do this for a living. I could see, sometimes, that I showed up in an answer. What I could never see was why that source got chosen over another, or what to change to show up more.

You’re optimizing in the dark, working from guesses and indirect signals, with no scoreboard to tell you if any of it landed.

It gets stranger. Ranking well in traditional search doesn’t guarantee you get cited in the AI answer. A page sitting at the top of Google can be completely absent from the AI’s response, while a lower-ranked page gets named again and again. The two systems don’t move together the way you’d expect. That means the playbook you spent twenty years learning doesn’t reliably transfer.

One habit makes every other flaw worse. People say “the AI” as if it were one thing. It’s several. ChatGPT is one machine. Gemini is another. Perplexity, Copilot, Claude, Grok, and whatever launches next month are all separate systems. Each of them also ships new versions that behave differently from the version before. You’re optimizing for a crowd of systems, and they don’t agree.

They pull from different sources. They weigh authority differently. They make things up at different rates. One leans hard on Reddit and reviews, another favors established publications, another prefers whatever it was trained on and barely looks at the live web at all. A change that makes you more visible in one can do nothing in a second and actively bury you in a third. There’s no single lever, no shared rulebook, no version of “do this and you’ll show up.”

You do the work. Then you watch it succeed in one place, miss in another, and backfire in a third, with no way to tell in advance which will happen.

For a business owner this is exhausting. You can’t pick one and ignore the rest, because your potential client might be asking any of them, and you have no idea which. So you’re forced to chase a dozen moving targets at once, each opaque, each slow, each wrong in its own way, none of them telling you the rules. That’s not a search engine you can learn.

That’s a field of them, all drifting, and you’re trying to be the right answer in every one at the same time.

I didn’t arrive at that from theory. I ran the exact same question, a request for a factual profile of me with sources, cold through every major engine on the same day. The spread wasn’t subtle. One of them, Copilot, pulled my current numbers, found the right material, and gave a clean profile with no hedging. Two others handed back a mix of current and years-old figures and openly flagged that they couldn’t reconcile them. One repeated a flat error I’ll get to in a moment.

And one, Gemini, generated an answer and then erased it in front of me before it finished, producing nothing at all. Five engines, the same question, the same day, and a range that ran from accurate to useless. There’s no “the AI” to satisfy. There’s a committee that doesn’t agree, and you have to win over all of them.

The flaw that surprised me most is speed. AI search is slow in both directions, and both directions hurt.

It’s slow to pick up new or corrected information, and slow is generous.

This one has beaten me for years. I’m a working professional whose facts keep changing. My book count goes up. My rates change. The kind of work I take shifts. That’s what a real career looks like from the outside: the facts move, because the person is alive and still working. The machines can’t handle it.

When I ran my profile test, the gap was right there in the numbers.

The current, correct figure for what I’ve written sat on my own pages, and one engine found it cleanly. The others reached past it and pulled a book count I passed years ago off scattered third-party pages, then served it up as current. Two of them showed the old number and the right number side by side and admitted they couldn’t tell which was true.

I’ve corrected my own source again and again. The dead figures keep crawling back into the answers, because they sit on enough other pages to outvote me. It’s the hardest problem I have hit in my own AI presence, and I haven’t solved it.

A static fact is easy. A fact that changes the way every living person’s facts change is something these systems are simply bad at holding onto.

And it’s slow to let go of bad information. The freshness problem cuts the other way too. Content you don’t keep current decays out of the answers, and stale errors can sit in the machine’s picture of you long after you’ve fixed the source. I track this on the living AEO updates page. So the machine punishes you twice: slow to be believed when you’re right, slow to be cleared when the machine is wrong.

The only real defense is to keep your home base current and consistent enough that, whenever the machine does finally update, it updates toward the truth. This slow rot has a name worth knowing, because drift is the failure mode almost nobody is watching for. This is the flaw that should worry a business owner most, and it’s the one people understand the least until it happens to them.

AI engines get things wrong. The errors are structural, not occasional. They generate confident, fluent, completely false statements, and the rate isn’t trivial. Studies put the rate of fabricated output anywhere from a few percent to more than a quarter of responses, depending on the topic. It climbs higher on niche subjects, which is to say on the lesser-known topics that describe most individual businesses and professionals. The more particular your niche, the more room the machine has to make something up about you.

Now the part with no good answer. When the machine invents something false about your business, there’s no edit button.

No form. No support line. No “report this” link that goes anywhere real. It states the falsehood to whoever asked, and you usually never find out it happened. This is the same wall as the stale book count, except now the wrong fact is invented. In my own profile test, three separate engines credited me with the same book. I didn’t write it.

It’s not mine. One engine making that mistake is a hallucination. Three engines making the identical mistake means a page somewhere in the crawled web has wrongly tied my name to that title. Every engine that found it repeated the error with total confidence. There’s nowhere to send the correction, no one to email, no lever to pull. I own my source and can make it say anything I want. The machines kept saying what they’d picked up somewhere else.

It’s worse than a simple mistake, because the machine invents things and also distorts the real data it holds about you.

This isn’t a hypothetical risk. A small solar company in Minnesota found that an AI engine was repeatedly and falsely telling people the state had sued it. The company tied real client cancellations and lost sales to it. They didn’t do anything wrong. The machine simply manufactured a damaging falsehood and told it to their potential customers, and there was no easy button to make it stop. Think about what that does to a business that runs on trust. That’s every business.

Can you appeal what AI search says about you?

Search had a feedback loop, clumsy as it was.

You could dispute things, file requests, work through a process. AI search effectively has none. There’s no court of appeal for what the model believes about you. You can’t challenge it, cross-examine it, or force a correction. The documented recourse, when there’s one at all, is a lawsuit. That’s an absurd and unaffordable answer for a small business that wants a falsehood taken down.

And don’t assume the company behind the machine will absorb the blame. The accountability tends to flow toward the business. When Air Canada’s own chatbot invented a refund policy that didn’t exist, the airline tried to argue it wasn’t responsible for what its bot said. A tribunal disagreed and held the company liable. The lesson cuts in an uncomfortable direction: “the AI said it” didn’t protect the business when the AI was its own.

You shouldn’t count on it protecting you when the AI belongs to someone else and is talking about you.

That’s the one that unsettled me, because it turns a passive problem into an active threat.

The same openness that lets you build authority with a strong home base also lets a determined bad actor poison the well. Researchers have shown this in controlled studies, and the results are sobering. By planting carefully constructed content where the machine would find it, they made the machine several times more likely to recommend a target product. In head-to-head tests, seeding the right material made the machine two to eight times more likely to choose one option over a competitor.

Worse, they showed you can attack in the other direction too: content engineered to disparage a rival, to label it unsafe or untrustworthy, can train the machine to suppress that competitor in its answers.

A competitor willing to spend the effort, or a stranger who simply wants to hurt you, can plant false signals built to inflate themselves and bury you. There’s no referee.

No one is checking. The machine swallows what it finds and repeats it with the exact same confidence it uses for the truth, and it can’t tell the difference. The system has no immune response. Anyone who tells you the AI playing field is fair is telling you something they haven’t tested.

It’s not fair, it’s not policed, and it can be bought.

If you want the wider context on how these systems get unreliable and how the errors compound, I’ve written more about the mechanics over on my fiction site, including a closer look at what the AI writing research says. The failure modes are real, measurable, and not going away on their own.

Watch: Google AI's Insane Answers, Matt Rose

Does AI search give you credit for your content?

There’s a quieter flaw that costs you in a different way.

The machine can absorb your content, your insight, the article you spent hours writing, and repeat the substance of it in an answer without sending anyone to you. You did the work. The machine delivered it. The person got what they needed and never learned your name, never visited your site, never had the chance to become a client. The credit, the click, and the relationship all evaporate into a summary.

You become an uncredited supplier to a system that gives you nothing back, and I think that’s the ugliest part of the whole arrangement. These companies built their products on work other people wrote and paid to host, and the business model only works if the reader never needs to visit the source.

Can you see which sources AI search uses?

And underneath all of it, you can’t see the inputs.

You don’t know which sources the machine weighed, which it trusted, which it ignored, which bit of stray nonsense it happened to pick up about you. You’re guessing at what the machine has read. That means you’re optimizing against a black box. Every other flaw on this list gets harder because of this one. You can’t fix what you can’t see, and you can’t see almost any of it.

Is AI search still worth the effort for a business?

If I’ve done my job, you’re now properly wary and wondering whether this is worth it at all. It is. I haven’t changed my mind from part one. AI search is where a growing share of decisions are getting made, and opting out isn’t a strategy. But you go in clear-eyed, and you build the one defense that works against this whole list of problems.

That defense is the same home base I spent the first part of this series arguing for. You can’t control the machine. You can’t make it fast, make it honest, make it fair, or make it accountable, and you can’t chase every one of the dozen engines on its own terms. What you can control is one thing: how consistently the truth about you appears across the web, anchored to a strong, accurate, owned source.

That single move is the one thing that works on all of them at once, because a clear, deep, consistent record is the closest thing there is to a signal every engine respects.

When the machine is slow, a consistent home base is what it eventually updates toward. When the machine is wrong, a corroborated record makes the falsehood harder to sustain. When someone tries to corrupt the picture, a deep and consistent footprint is what’s hardest to drown out.

Every one of these flaws punishes businesses with thin, scattered, inconsistent presences, and rewards the ones who built something solid early and kept it current.

AI traffic is worth pursuing, and the businesses that win at it won’t be the ones chasing a hack or buying a service that promises to game the system.

I’d steer any client away from those services, because they’re selling certainty about machines nobody can see inside. The winners will be the ones who built something real, made it the clearest and most authoritative source about themselves, and made it easy for the machines to get them right. The flaws are exactly why that work matters.

In a system this opaque, this slow, and this corruptible, being the strongest honest signal in the room is the only move that holds up. That is what I tell clients about AI when they ask whether they should worry about it.

Frequently Asked Questions

From the author

The Day Your Website Died
Search is splitting into dozens of AI engines. In The Day Your Website Died I cover answer engine optimization and what to do when your traffic changes shape.