AI Retrieval Changes the Web’s Bargain

By /

For a long time, the open web ran on an imperfect but understandable bargain.

Publishers put useful things online. Search engines crawled those pages. In exchange, search sent at least some people back to the source. The deal was never pure charity. Search engines got an index. Site owners got discovery, reputation, revenue, leads, community, or at least the faint comfort of watching analytics graphs twitch in a way that suggested humans still existed.

That bargain is changing.

AI retrieval, answer engines, and RAG systems do not always behave like old search. They may visit a page, extract the useful part, summarize it somewhere else, and satisfy the user without a click back to the original source. Sometimes that is great for the user. Sometimes it is rough for the person or organization that did the work.

The sane response is not to declare the web dead, board up the windows, and start writing manifestos by candlelight. Tempting, but no.

The better response is to make the bargain explicit. If machines are going to read, summarize, cite, train on, or retrieve from the web, publishers need better ways to say what is allowed, what deserves attribution, and where access should stop.

Search crawled for discovery. Retrieval crawls for answers.

Traditional search crawlers were built around discovery. Googlebot visits a page, indexes it, and later shows a result that points users back to the source. That model has plenty of problems, but at least the direction of travel was legible: crawl, rank, refer.

AI retrieval changes the shape of the trip.

A RAG system can retrieve a paragraph, use it as context, and answer the user inside another product. An AI search result can summarize a page with a source card nearby. A browser agent can pull facts from multiple sites and produce a combined answer. A training crawler can gather material for future model behavior with no immediate reader involved at all.

Those are different uses, and lumping them together makes the conversation worse.

A tool that cites a source while answering a question is not the same as a scraper that ignores crawler rules and quietly builds a resale dataset. A search assistant that sends qualified readers to a site is not the same as an answer box that replaces the visit entirely. The web needs room for those distinctions.

But publishers are not imagining the pressure. Digiday reported in 2025 that AI search referrals were growing but still not enough to offset the damage from zero-click search, with one analysis finding that nearly 69% of news-related searches ended without a click. The exact number will move around by market and method, but the direction is not subtle: more answers are happening away from the original page.

That is the part worth taking seriously.

Robots.txt was a handshake, not a contract.

The old control surface for crawlers was robots.txt, a plain text file at the root of a website. It is beautifully simple. It is also limited.

robots.txt tells cooperative crawlers what they should avoid. It does not enforce access. It does not negotiate payment. It does not distinguish cleanly between search indexing, AI training, summarization, citation, archiving, and commercial reuse. It definitely does not stop a bad actor who decided manners were too expensive this quarter.

That does not make robots.txt useless. It still matters because many major crawlers respect it, and because explicit policy beats mystery. But it was designed for crawler etiquette, not the full economics of AI retrieval.

That is why more sites are tightening access. Cloudflare’s 2025 crawler analysis found that AI crawler traffic was growing quickly and that a meaningful share of top domains were already blocking AI crawlers. Publishers are not being irrational. They are reacting to a system where the old rewards are less reliable and the new uses are less clear.

The problem is that blocking everything is a blunt tool. It protects the door by making the house harder to visit. Sometimes that is necessary. Sometimes it cuts off the kind of discovery a publisher actually wants.

This is where the web needs better controls than “come in” or “go away.”

The new rules are starting to appear.

The interesting work now is not just blocking crawlers. It is building ways to describe terms.

Cloudflare has been experimenting with crawler controls and introduced pay per crawl, a model where site owners can charge AI crawlers for access. That is not a universal solution, and it will not magically fix the economics of publishing by Tuesday. But it points in the right direction: access to useful work has value, and that value should not be assumed away because a bot did the reading.

RSL, or Really Simple Licensing, is another attempt to make licensing terms machine-readable. The idea is straightforward: publishers should be able to say, in a form automated systems can understand, whether content can be used for search, AI training, summaries, or other kinds of reuse.

Then there is llms.txt, a newer convention described at llmstxt.org. It is not a magic enforcement layer. It is closer to a guide for AI systems: here is what matters on this site, here are the useful source pages, here is how to understand the content without chewing through every navigation element and footer link like a raccoon in a dumpster.

For provenance, standards like C2PA Content Credentials are trying to preserve information about where media came from and how it was made. That matters more as content gets copied, summarized, transformed, and re-shared through systems that can easily strip away context.

None of these standards are done. Some will become infrastructure. Some will become footnotes. Some will become yet another thing consultants put in a slide deck while saying “ecosystem” with a straight face.

Still, they all point to the same need: the web needs clearer machine-readable signals for ownership, attribution, access, and reuse.

Clear websites will age better than mystery boxes.

The practical takeaway for site owners is not “install one file and defeat AI.” Sorry. The universe remains committed to being annoying.

The practical takeaway is to make your site easier to understand and harder to misrepresent.

Start with the basics:

  • Keep author names, publication dates, update dates, and source links visible.
  • Use structured data where it helps, especially for articles, authors, organizations, citations, and licensing.
  • Keep important pages easy to find without making a crawler solve a maze.
  • Decide which crawlers you allow, which you block, and why.
  • Watch server logs occasionally so you know who is actually hitting the site.
  • Consider llms.txt if you want to guide AI tools toward your best source pages.
  • For original media or research, keep provenance and licensing information attached when you can.

This is not just an AI search trick. It is good publishing hygiene.

I made a related point in AI Search Rewards Clear Websites, Not Dead Ones: a site that is clear for machines is usually clearer for humans too, as long as you do not let structured data become a substitute for actual usefulness. The goal is not to build a website for bots. The goal is to stop making your work unnecessarily easy to detach from its context.

Human value still matters.

There is one uncomfortable editorial truth in all of this: if a page can be fully replaced by a two-sentence summary, it probably will be.

That does not mean every page needs to be a novel. It means publishers need to be honest about what the page is doing. Commodity answers are easy to compress. Original experience, careful judgment, specific examples, useful tools, strong sourcing, and a real point of view are harder to flatten.

That is why AI-era publishing cannot just be a metadata exercise. Metadata helps retrieval systems understand a page. It does not make the page worth understanding.

The stronger move is to publish work that has a reason to exist beyond filling a search result. Show the test. Name the tradeoff. Link the source. Explain what changed your mind. Put the operational lesson somewhere a reader can actually use it. I use that same standard when evaluating tools after the demo in How I Evaluate AI Tools After the Demo. The thing that survives contact with reality is usually more valuable than the thing optimized for a pretty summary.

AI retrieval is not going away. It is too useful. People like getting answers quickly, and pretending otherwise is just nostalgia with worse UX.

But usefulness does not erase the need for attribution, consent, provenance, and fair terms. The old web bargain was implicit. The next one needs to be visible.

If you publish online, the work now is simple, not easy: make your site worth reading, make your sources clear, make your rules explicit, and do not assume the old search bargain still protects you just because it used to.

Read next

Scroll to Top