Subscribe
Analysis
6 min read

No Free Crawl: What AI Access to the Web Actually Costs in Late 2026

Training crawlers now drive half of all crawler requests, September's default-block rule redrew the line, and three price mechanisms for AI web access have emerged. Only one of them reliably pays.

Olivetti brochure style illustration of a crawler robot handing a coin across a toll gate to a printing press, headline reading NO FREE CRAWL
Olivetti brochure style illustration of a crawler robot handing a coin across a toll gate to a printing press, headline reading NO FREE CRAWL

For thirty years the web had one deal with automated visitors: robots.txt and hope. Search engines crawled, publishers allowed it, visitors came back. The exchange was loose but real. AI crawlers broke it, and 2026 is the year the replacement arrived: not one new deal, but three different price mechanisms that mostly do not pay what the headlines suggest.

The numbers that ended the old exchange

Cloudflare sits in front of more than 20% of the web, so its accounting is the closest thing to a census. In a report published this month marking one year since its "Content Independence Day" policy shift, the company says more than 50% of internet traffic is now non-human, a majority it describes as the first in internet history. Of crawler requests it can classify by purpose, 52% were for AI training as of June 2026, up from 22% in spring 2025. Mixed-use crawlers that blend search, agent fetches and training account for over 36% of activity.

The referral side is what made publishers angry. Cloudflare's data shows that for every hour people spend searching for information online, only 15 minutes are now spent on the open web. In the most heavily crawled content categories, human traffic fell by as much as 40% in less than a year. Third-party analyses of the same Radar data put the asymmetry in blunt terms: SEOmator calculated that ClaudeBot crawled roughly 23,900 pages for every referral visit it returned in the first quarter of 2026. Training crawlers, by design, return none.

Cloudflare's own summary of the problem: content is still crawled, indexed and used, "increasingly without corresponding traffic being returned to the source." Some publishers now plan for what they call "Google Zero," a world where search referrals amount to nothing.

What changed on September 15

On July 1, Cloudflare replaced its single "block AI bots" switch with three independent controls, one per declared behavior: Search (indexing that is supposed to send readers back), Agent (real-time fetches on a person's behalf) and Training. From September 15, its defaults changed: Training and Agent bots are blocked by default on pages that display ads, for new customers, new sites and all free-plan domains, with opt-out available.

The rule with the widest consequences targets mixed-use crawlers. Cloudflare now judges a crawler by the most restrictive behavior it performs, and its announcement names Googlebot, Applebot and BingBot as multi-purpose crawlers that get blocked where Training is blocked. TechCrunch framed the whole change as a deadline: AI companies have until September 15 to separate their crawlers by purpose, or risk being shut out of a fifth of the web by default. The uncomfortable implication for publishers is that a "no training" setting on ad pages can quietly evict the classic search crawler too.

Where the money actually is: bilateral deals

Despite the machinery described below, the large sums in this market are still ordinary contracts. Cloudflare counts more than 50 publisher-AI licensing agreements signed since 2023. The reported figures cluster at the top of the market: News Corp's deal with OpenAI at roughly $250 million over five years, a separate News Corp arrangement with Meta reportedly worth up to $50 million a year, Reddit's agreements with Google (about $60 million a year) and OpenAI (an estimated $70 million), and Amazon's deal with The New York Times at a reported $20-25 million a year. The one verified floor comes from corporate filings: IAC disclosed at least $16 million a year to Dotdash Meredith. Almost everything else is undisclosed.

Two reference points bracket the rest of the market. Anthropic's $1.5 billion copyright settlement in September 2025 worked out to roughly $3,000 per work and set a de facto baseline authors' representatives quote from. Early OpenAI pitches to smaller publishers reportedly ran $1-5 million a year, and industry trackers are candid that the long tail of small and mid-size sites will see no deal at all. Licensing pays for brand-name corpora with negotiating leverage, not for the web as a whole.

The tollbooth: pay per crawl mostly declines

The machine-readable version of the deal is Cloudflare's pay per crawl, in private beta since July 2025.

Illustration of crawler robots queueing at a toll gate on a highway of paper pages

A site owner sets a price per zone. A crawler that wants the page sends a payment-intent header and gets an HTTP 200 with a crawler-charged header; one that doesn't gets an HTTP 402 Payment Required with a crawler-price header. Cloudflare acts as merchant of record, aggregates the billing events and pays publishers. In principle, robots.txt's please go away became a price list.

In practice, almost nobody pays per fetch. As of April 2026, third-party analysis of Cloudflare's disclosures counts more than 1 million enabled customers and over 1 billion daily 402 responses across the network, but estimates the actively monetizing subset in the low tens of thousands, and finds the large majority of 402s are simply declined. Engagement concentrates in a handful of bot identities: ChatGPT-User and OAI-SearchBot transact some paid fetches, PerplexityBot pays for select premium sources, while ClaudeBot, Google-Extended and Bytespider largely walk away from priced URLs. Publisher pricing has gone bimodal for rational reasons: a large group charges $0.001-0.005 per fetch and gets volume, a smaller group charges $0.05-0.25 and gets picked off only by premium-content buyers, and the middle band earns nothing because it attracts neither.

Stack Overflow, which adopted the model with Cloudflare in February 2026, describes it less as revenue than as leverage: a way to put publishers "back in the driver's seat" while a real market forms around protocols like x402 for machine-to-machine payment.

From fetch to use

The newest mechanism moves the billing unit from the crawl to the consumption. Cloudflare announced in July that pay per crawl is evolving into a pay-per-use model that pays when content shows up in a partner's AI answers, with Ceramic.ai and You.com named as pilots. It is candidly an experiment: pricing the outcome matches payment to value, but is much harder to measure and verify. On the product side, Perplexity's Comet Plus passes 80% of subscription revenue to publishers whose content its users access, and startups like ProRata split revenue by which sources an answer drew on. Nobody has converged on an auditable standard for what a "use" is.

The Google-shaped hole

Every one of these rails has an exception carved around Google. It still accounts for approximately 88% of referral traffic, and it crawls for search and its AI features with the same bot, which, per Cloudflare, gives Google access to about twice as much information as leading AI companies while making a publisher's training opt-out also an indexing risk. The 2026 restructuring of the crawler economy is, so far, a market for everyone except the company that created the old one.

What the price tag means

Three honest conclusions. First, content acquisition is no longer free for AI products: unit economics that assumed a $0.001-per-page scrape now face priced fetches, licensed-only corpora or blocking, and labs are responding by buying selectively rather than crawling indiscriminately. Second, the blocking tools' declared purpose is leverage, not revenue. Cloudflare's chief strategy officer says as much, and the pay-per-crawl data bears it out: the tollbooth mostly collects refusals, while the refusals push both sides toward contracts. Third, the long tail is still uncompensated. Between the settlement math of $3,000 per work and the tens of thousands of publishers actually earning per-fetch fees, the "market for web content" remains, like the licensing market before it, a top-100 phenomenon.

The old deal was: crawl for free, maybe send a reader back. The new deal is: declare your purpose, or pay, or leave. Which of the three an AI company chooses is now itself a piece of competitive intelligence worth reading.

Sources

Analysisweb economyAI Agents