The Web Got Fenced: What AI Search Costs Small Sites

The web was built so anyone could publish and anyone could find it. Nobody had to be vouched for.

That property was never written down as a principle. It was a side effect of how discovery worked: a volunteer directory editor, a ringmaster, a crawler following a link. Every one of those would consider a page nobody had heard of. It lasted about twenty-five years.

It is gone, and the thing that replaced it asks a different question. Not whether a page is good. Whether anyone will vouch for it.

I watched a smaller version of this happen once before, from inside. I ran web rings through the 1990s and was there when a corporation took the system apart and the community fought back for years. We won that fight. Rings died anyway, because the road moved. That is the pattern worth carrying into what follows.

A link used to be a vote

PageRank had a democratic story attached to it, whatever its faults. A link was a vote, and a small site with something good could accumulate votes from people who found it useful. The system was gamed constantly, and Google spent twenty years fighting that. The theory still pointed at merit.

It also had a property people underrate now. A link cost nothing to give. Any page could vote for any other page, and a page with no budget could receive a vote from a page with an enormous one. Money helped, because money bought content and outreach and eventually links outright. Money was never the only currency.

Nothing about the current model points at merit. The requirement is corroboration: other entities saying who you are, across platforms that mostly charge, or reward volume, or both. Yelp. Trustpilot. G2. Capterra. Clutch. Trade directories. Press wires. An AI system reads the consensus across those sources and repeats it.

Sit with the list for a second. Every one of those is a gate somebody else operates, and most of them collect rent at the gate. That is a spend filter wearing the costume of a trust signal.

How does AI search decide which websites to cite?

Mostly by looking at what other sites say about you, and only secondarily at what your own site says. The industry has a phrase for it, off-page corroboration, and the practitioners who get results are open about the mechanics. Your own page can claim anything. A page you do not control saying the same thing carries weight your own claim cannot.

In practice that means a specific shopping list. A listing on the review platform that dominates your category. Reviews on it, in volume, recent enough to look alive. A presence in whatever directory the model reaches for when someone asks about your field. Mentions in publications the system already trusts. An entity record it can attach all of this to, so the mentions resolve to one identity instead of scattering across a dozen half-matches.

Then the same claim, phrased the same way, repeated across all of it. The consistency is the point. A machine reconciling forty sources is looking for agreement, and variation reads as uncertainty.

I listened to two well-known search professionals debate this recently, and the frank part of the conversation was the money. One described a client whose monthly retainer went from ten thousand dollars to seventy thousand for the same category of work under a newer label. That is the market pricing the gates.

The case for corroboration is stronger than I want it to be

I should argue the other side properly, because the engineering logic is sound.

A system that answers questions in prose carries a burden search results never did. A list of ten blue links pushes the judgment onto the reader, who can see the source and decide. An answer in a paragraph has already decided. Whatever it says becomes what the person believes, and there is no second opinion visible underneath it.

That raises the cost of being wrong enormously. A search engine that ranks a bad page has shown you a bad page. A system that synthesizes a bad page has told you something false in its own voice. The pressure to be right pushes hard toward sources that many independent parties agree on, because agreement is the only proxy for truth that a machine can compute cheaply.

People reason the same way. A stranger recommends a plumber and you weigh it lightly. Four unconnected neighbours recommend the same plumber and you call him. Nobody thinks that is unfair when a person does it.

The difference is what agreement costs to obtain. Your neighbours are not selling their opinions. The platforms sitting between a small business and its corroboration are running a business, and the price of admission is set by them.

Every gate collects a toll

Look at what a full corroboration footprint requires and the shape becomes clear.

Review platforms are the obvious tier. Most let you claim a listing free and then charge for anything that makes the listing perform. Directories in professional fields often charge for membership before the listing exists at all. Industry association listings usually require dues. Press coverage is either earned through relationships that took years, or bought through wire services that will publish almost anything for a fee.

Awards deserve their own paragraph. A meaningful share of the awards that show up as corroboration are pay-to-enter, and some are pay-to-win in everything but name. They exist because a badge on a website and a mention in a listing feed the consensus. The machine has no way to tell a juried prize from a purchased one, so both count.

Add these together and the cost of being visible is no longer the cost of making something good. It is a recurring subscription to being believed. A one-person business with a genuine specialty pays the same tolls as a funded competitor, from a fraction of the revenue, and the funded competitor can simply buy more of the same signals.

The index got smaller, and that matters more than the tactics

Something structural sits underneath the corroboration question, and it may matter more.

One practitioner in that conversation described watching a site receive a few hundred clicks from a search engine while the same nine pages were fetched tens of thousands of times as citations. His read was that these systems are not searching the live web for each answer. They build a working table from a small slice of the index, then read from the table.

I have no way to verify that from the outside, and he presented it as an observation from his own data. If it is even roughly right, the consequence is severe. A site outside that slice never enters a ranking contest at all. Nothing reads it. There is no page two to climb to.

The economics make it plausible. Indexing the whole web is expensive and refreshing it constantly is worse. If a small fraction of pages answers the overwhelming majority of questions, then building and maintaining only that fraction is the rational engineering choice. Nobody has to decide to exclude the model train page. The budget decides.

There is a second layer that behaves differently. What a model absorbs during training is not the same as what it fetches while answering. Training gives it language, general facts, and the brands repeated so often they became ordinary nouns. Retrieval gives it whatever is current. A small site has almost no path into the first and a narrow, contested path into the second.

Why does manufactured consensus work on AI search?

Because the system rewards what is corroborated, and corroboration can be built. This is the second-order effect, and it bothers me more than the unfairness.

The same conversation covered the working methods openly. Press release blasts across four or five wire services. Force-indexing the resulting URLs. Buying cheap pop-under traffic through a redirect link to fake a virality spike, six thousand hits in two hours to push a page up. In the gambling vertical, deliberate poisoning of what the models hold about a competitor. One participant paused twice to say search engineers might be listening.

The demonstration that stuck with me was smaller and much more revealing. Two of them ran an experiment where one published a press release claiming he had replaced the other as the leading figure in his field. Within hours the AI systems repeated it as fact. The other spent the rest of the day publishing counter-material, and by evening the story had flipped back. Nothing true had changed. A title with no meaning moved twice in twelve hours because two people took turns feeding the machine.

None of that improves anything for a reader. It is the skill the incentives select for. When visibility depends on consensus and consensus can be purchased, the market prices consensus and stops pricing quality. I wrote about the structural cracks in AI search from a different angle, and this is the same fault line seen from above.

Who loses when AI search replaces search engines?

Abstraction hides the specific casualties, so I will name them.

The hobbyist with twenty years of accumulated knowledge about a narrow subject. No business, no reviews, no directory, no reason for any platform to list him. His pages are the best material on the internet for his topic and there is no mechanism by which a model finds out.

The small nonprofit whose entire budget goes to the thing it exists to do. Every dollar spent on corroboration infrastructure is a dollar not spent on the work, and the funded organizations in the same field have staff for this.

The regional expert whose reputation lives in conversations, referrals, and a professional community that never wrote anything down. Corroboration exists in abundance and none of it is machine-readable.

The academic or the retired practitioner publishing out of interest instead of commerce. No incentive to build a commercial footprint, so no footprint, so no citations.

What connects them is that the thing they have is knowledge, and the thing the system measures is commercial presence. Those overlap sometimes. They are not the same, and nothing in the current architecture can tell the difference.

The loop closes on itself

Then it compounds, which is the part that makes it hard to reverse.

A site that gets cited gets read. Being read produces mentions, and mentions produce more corroboration, which produces more citations. Meanwhile the site nobody cites gets no traffic, generates no mentions, accumulates no signals, and slides further from the slice that gets read at all.

Search had this problem too, and it was real, but a link from anywhere could break the cycle. One well-placed mention could pull a site into visibility. In a corroboration model the entry requirement is a set of platform presences, and platform presences do not arrive by accident. Somebody has to build them deliberately, which means somebody has to fund them.

The gap between the visible and the invisible stops being a gradient and becomes a wall.

What can a small website do about any of this?

Less than I would like, and more than nothing at all.

Be legible. State plainly who you are, what you do, and who you serve, on a page built for that job. Machines doing the indexing run cheap and do no reasoning, so a conclusion you leave implicit never gets drawn. If your credentials imply expertise, write the sentence that says so. The inference will not be made for you.

Be consistent. The same description in the same words everywhere you appear. A machine reconciling your identity across sources treats variation as uncertainty, and uncertainty as a reason to cite somebody else. This costs nothing and most sites get it wrong.

Take the corroboration that is free. Professional association listings you already pay dues for. Directories in your field that do not charge. Interviews, podcasts, and mentions you earn by being worth talking to. That is the same mechanism the paid version buys, earned slowly and cleanly.

Answer questions in the form people ask them. Headings phrased as the actual question, answered directly underneath in a passage that stands alone. A page written as one long argument gives a retrieval system nothing to lift. A page with clear answers to real questions gives it something to quote.

Build the thing that cannot be purchased. A body of specific work that exists nowhere else is the one asset the machinery cannot manufacture. Original research, a documented method, an archive built over years, an account of something you lived through. It will not put a small site above a funded one. It gives that site a reason to be cited when it does surface, which is what a citation is now worth.

Refuse the fabrication playbook. Not for moral credit. The wire blasts and the traffic spikes are calibrated to whatever detection looks like this quarter, which is why the people selling them keep selling the next version. You would be renting a position and paying rent forever.

The loss is real even if you can work around it

I can operate inside this system. I have spent considerable time getting my own site legible to it, and I will spend more. Being able to compete within a set of rules does not make the rules fair, and it does not make the loss smaller.

What went away was a web where discovery did not require permission. A page could exist because one person cared, and it could be found by another person who cared, and no intermediary had to agree that either of them counted. That property was strange and valuable and mostly accidental, and it is gone.

Something else went with it. The old system, for all its gaming, at least aimed at a question with a real answer: is this page useful. The new one aims at a question with a purchasable answer: does anyone vouch for this page. Those look similar from a distance. They select for completely different things, and only one of them rewards doing the work well.

In 2000 I thought the enemy was a corporation that did not know it was in a fight, and I was wrong about that too. The corporation was the visible part. What ended web rings was that the road moved, and after it moved there was no reason to drive the old one.

The road has moved again. Before you spend money trying to get onto the new one, read what the AI visibility industry is selling and what the evidence says about it, because most of it does not work. Then start with the AI writing hub for the practical side, and look at what I do with AI and content if you would rather not work it out alone.

Frequently Asked Questions