THE ENCLOSED WEB · A THREE PART SERIES, PLUS THE SOURCE
1Black Tuesday2The Web Got Fenced3What the Evidence SaysThe Original 2000 Page
Everyone selling AI search optimization is selling it into a vacuum. Nobody publishes a study. The advice arrives as confident assertion, and the people giving it are rarely the people ranking.
Some of it is measurable, though, and the measurements are unkind. What follows is what the numbers say, what got three real companies destroyed, and the short list of things that survive scrutiny.
Does an llms.txt file help a website get cited by AI?
This was supposed to be the answer. Publish a plain text index of your best content at a fixed address, and AI systems would read it instead of wrestling with your HTML. It costs about twenty minutes and the proposal is sensible on paper.
The measured reality is different. One analysis covering more than five hundred million AI crawler events over ninety days found a few hundred requests for that file in total. The major crawlers fetch HTML pages the way search crawlers always have.
A survey of three hundred thousand domains put adoption around one site in ten. Among the fifty most-cited domains, exactly one had the file.
Sit with that last number. The sites winning at AI citation are, with one exception, not doing the thing being sold as the way to win at AI citation.
Google has said so on the record. Its position is that it does not use the file and has no plans to, one of its search staff compared it to the discredited keywords meta tag, and its own guidance lists it among unnecessary tactics.
Publish one if you like. It costs almost nothing. Do not mistake it for a strategy, and be careful with the version some people recommend where every page gets its own markdown copy. If those files are indexable you have just created duplicate content at scale, which dilutes crawl budget and can suppress the original pages.
What the tactics that do work will cost you
There is a set of methods that demonstrably moves AI visibility, and practitioners describe them openly at conferences.
Press release blasts across four or five wire services. Force-indexing the resulting URLs. Buying cheap pop-under traffic through a redirect link to fake a virality spike, six thousand hits in two hours to push a page up. In some verticals, deliberate poisoning of what the models hold about a competitor.
The demonstration that stuck with me was smaller and more revealing. Two well-known practitioners ran an experiment where one published a press release claiming he had replaced the other as the leading figure in his field. Within hours the AI systems repeated it as fact. The other spent the day publishing counter-material, and by evening the story had flipped back. Nothing true had changed. A title with no meaning moved twice in twelve hours because two people took turns feeding the machine.
Google has been writing policy against exactly this. In May 2026 it extended its spam policies to explicitly cover attempts to manipulate generative AI responses in Search. In April it added a rule on back button hijacking. Its site reputation abuse policy, aimed at third-party content published mainly to benefit from a host domain’s ranking signals, has been named in the rater guidelines since January 2025.
Three collapses worth studying
Abstract policy is easy to ignore, so look at what happened to three specific companies.
Forbes Advisor ranked number one for best CBD gummies and number two for how to get rid of roaches. None of it connected to business journalism, and it was not run by Forbes: a separate company operated it, with Forbes owning around forty percent. More than twenty million monthly search visits built in under four years. After September 2024 it fell to roughly 2.9 million a month, about twelve percent of peak.
HubSpot’s blog was the case study everyone cited, including me. It ranked for famous quotes, resignation letters and how to type the shrug emoji, none of which had anything to do with what HubSpot sells. It later acknowledged a seventy-five percent reduction in English language blog traffic. Its own marketing chief wrote the lesson afterwards: you need a differentiated point of view, differentiated data, and a clear reason you are uniquely qualified.
Tailride is the frightening one because it was small and fast. An invoicing startup published 22,000 AI-generated pages programmatically, scraping sources and publishing with no human review. Return policy pages for retailers, four thousand Swift code pages, five thousand financial definition pages, most of it unconnected to the product. It worked for three months. On 28 February 2025 clicks from Google went to zero overnight, and the index fell from over twenty thousand pages to one over the following six months. No manual action and no warning.
The founder wrote the whole thing up publicly, which took some nerve, and his conclusion is the useful part: they had not built trust, and to Google they probably looked like a spammy site trying to game the system. The AI was never the violation. Thin repackaged pages published at scale to capture rankings was the violation.
What do these three failures have in common?
Each stretched the distance between what it had earned trust for and what it was publishing.
Forbes borrowed authority it had not earned. HubSpot drifted away from authority it had. Tailride manufactured authority at a scale no human was checking. Three different roads to the same place.
That gives you a test that costs nothing to apply. Is every section of your site about something you are qualified to cover? Are you hosting anyone else’s content mainly because your domain ranks? If a machine helped produce a page, did an accountable person read and improve it before publication? Would you put your name on this page and be proud of it on a screen at a conference?
Tailride’s founder admitted his 22,000 pages could not pass the last one. Most sites have a few that cannot either.
The one documented recovery
Examine.com is an independent health and nutrition research organization, and it is the only case here that shows a way back.
It lost roughly ninety percent of its organic traffic in 2018 during a core update that hit health sites hard. It did not change the research. It changed how clearly the site demonstrated the process behind the research: named researchers and fact-checkers, stronger editorial policies, clearer funding disclosures, better about pages. Traffic recovered past a million visits a month.
Then it got hit again in 2024, falling from around 575,000 monthly visits in January to roughly 226,000 by December. It kept publishing research summaries and kept the credentialing work in place, and by September 2025 it was over 750,000 a month, higher than before the drop.
This does not prove expertise makes a site immune. It clearly did not. It shows that a credible publisher recovers by holding its mission and improving how visibly it demonstrates its process, which is a slower and less exciting answer than anything being sold.
What survives scrutiny
What follows is short, and none of it is proprietary.
Be legible. State who you are, what you do and who you serve, plainly, on a page built for that job. Indexing systems run cheap and do no reasoning. A conclusion you leave implicit never gets drawn, so if your credentials imply expertise, write the sentence.
Be consistent. The same description in the same words everywhere you appear. A machine reconciling your identity across sources treats variation as uncertainty, and uncertainty is a reason to cite somebody else.
Show the process, not the conclusions alone. That is what Examine changed and what HubSpot’s own postmortem described. Named people, stated method, disclosed funding, visible editorial standards.
Collect the corroboration that is free. Professional association listings you already pay dues for. Field directories that do not charge. Interviews and mentions you earn by being worth talking to. Local sponsorships you would have supported anyway.
Answer questions in the form people ask them. Headings phrased as real questions with a passage underneath that stands alone. A page written as one long argument gives a retrieval system nothing to lift.
Write the attributes people search on. Nobody types a category name into a chat window. They describe a situation. If your page does not contain the words describing that situation, you are not a candidate for the answer.
Say who you are not for. This is the least obvious one and possibly the most useful. Telling a model who your work does not suit helps it recognize when your work does suit. Most businesses hide exactly this information.
The honest summary
The AI visibility industry is selling certainty about a system nobody has documented, and its most popular product is contradicted by the only large-scale measurement anyone has published.
Underneath the noise, the durable stuff is the same work it has been for years: be clear about who you are, demonstrate how you know what you know, earn mentions from people with no reason to flatter you, and build something specific enough that nobody else could have produced it.
That is slower than a wire service package and it does not come with a dashboard. It also cannot be taken away by a policy update, which the three companies above would have appreciated.
For the underlying argument about why visibility now depends on being vouched for, and what that costs the small web, read what AI search costs small sites. For a longer view of how this has happened before, I was inside the last version of it. And if you would rather have someone build the durable version with you, that is what I do with AI and content.
The Guides That Get Your Book Written, Published, and Sold
Four short, practical guides on writing, publishing, and selling your book, plus the occasional note when there's something worth your time. No fluff, no daily inbox clutter. Drop your email and they're yours.
We use MailerLite to manage our list and send these emails. Your address is used only to send you what you signed up for. We will not sell it, share it, or use it for anything else, and you can unsubscribe anytime.
