A website can now do what a paper can: hand a machine a table of contents. The convention is called llms.txt, Jeremy Howard proposed it in September 2024, and by October 2026 BuiltWith counted it on 14,586,789 live websites, up from 844,473 a year earlier. That part of the story is settled: publishers showed up.
The other half is the awkward one. In Ahrefs' June 2026 study of server logs across roughly 137,000 domains, 97% of llms.txt files received zero requests from AI crawlers during the study month. Of the requests that did arrive, AI retrieval bots made up about 1.1%. SEO audit tools accounted for roughly 22%, and the rest came from unidentified bots and general web crawlers. The thing most likely to fetch your llms.txt is somebody's SEO scanner.
A separate analysis of more than 500 million monitored AI bot visits over a 90-day window reached a similar place from another direction: only around 400 of those visits targeted an llms.txt file directly. GPTBot, ClaudeBot, PerplexityBot and Google-Extended overwhelmingly fetched HTML instead.
Publication is up 17x year over year. Reading is approximately nil. That gap is the actual state of the standard in late 2026.

What the file is supposed to do
llms.txt is a Markdown file at a site's root that lists the pages the owner considers most important, each with a one-line description, and can link to clean Markdown versions of those pages. The pitch was that a model or agent trying to understand a site should get a curated index instead of scraping navigation bars and cookie banners. Howard framed it in the original proposal as solving the context problem: site authors know which pages matter, so let them say so.
The idea caught on in a way few web conventions do. It reached version 2 in August 2026, which allows an index at any path rather than only the site root and formalizes ways for an HTML page to advertise its Markdown alternate. Chrome's Lighthouse added an optional llms.txt audit to its agentic browsing checks. WordPress plugins, static site generators and documentation platforms added generators.
None of that machinery matters unless a reader shows up. Few have.
What the studies found
Three independent datasets now describe adoption and use, and they disagree in an instructive way.
SE Ranking analyzed 300,000 domains and found an llms.txt adoption rate of 10.13% after eighteen months of industry conversation. The same study tested whether publishing the file correlated with how often a domain gets cited by large language models. Both statistical and machine-learning analysis found no relationship. The researcher Ethan Smith reported the same null from a different angle: in a random sample of about 50,000 URLs cited in AI answers collected for research, not a single one was an llms.txt file.
Ahrefs measured requests rather than presence and found the 97%-never-fetched result described above, alongside adoption growth of 8.8x year over year on its sample.
BuiltWith counts installed files and shows the growth curve from under a million sites to nearly 15 million. It measures the presence of the file, not whether anything reads it, which is exactly where the discrepancy comes from.
The three numbers together describe a market with enthusiastic supply and almost no demand.
Google closed the SEO question
Google removed any ambiguity about rankings. Its guidance for AI features in Search, updated in May 2026, states that site owners do not need to create new machine-readable files, AI text files, markup or Markdown to appear in Google Search, including its generative AI capabilities, because Google Search itself does not use them. There is no boost and no penalty for having the file, in classic results, AI Overviews or AI Mode.
Google's John Mueller went further in public commentary, comparing llms.txt to the old keywords meta tag: a file bots do not request and services do not use. Asked why some Google developer properties publish one anyway, he said an internal content system had added support and some teams had not removed it, not that Search consumes it.
No AI lab has publicly committed to reading llms.txt as a ranking or citation input in production either. OpenAI documents separate crawlers for search, training and user-directed visits and lists no llms.txt requirement for ChatGPT Search inclusion. Anthropic similarly publishes indexes for its own docs without promising that Claude reads everyone else's.
The one place it genuinely works
The skeptical data flattens one real finding: developer tooling reads the file, because developers tell it to.

The convention started in documentation circles and still works there. OpenAI, Anthropic, Cloudflare and Stripe all publish llms.txt indexes for their developer docs. Mintlify, a documentation platform, generates llms.txt and llms-full.txt automatically for every project it hosts, which is part of why BuiltWith's count exploded: platform-generated files ride along with every new docs site whether or not anyone edits them.
When a coding agent is fetching a library's authentication guide, a curated Markdown index beats parsing a JavaScript-rendered docs site. The user can paste the index URL into the agent, and agents that browse documentation follow it. This is a present-tense, narrow benefit: it applies to documentation-heavy, API-first sites whose audience works inside coding assistants.
There is a second, newer signal in the same direction. Shopify added native storefront routes for /llms.txt and /agents.md, generated automatically for stores, as part of its agentic commerce push. Wix generates one for eligible sites alongside structured data and its site MCP server. Neither company spent engineering budget on a crawler statistic. They are betting that shopping agents will browse catalogs, and they want merchants readable if that day arrives.
What publishers should conclude
The honest summary is unglamorous. If you run documentation or an API, publish the file and automate its upkeep, because coding agents are real users today. If you run an online store on Shopify or Wix, you already have one whether you looked or not. If you have a small marketing site and the file would be hand-maintained, skip it. A stale index that nothing fetches is worse than no index, and no platform penalizes you for its absence.
What you should not do is buy anything that promises llms.txt will improve your AI citations. Across 300,000 domains nobody has found the correlation, and across 50,000 sampled AI answer citations the file never appeared as a source. The vendors selling "AI visibility" through llms.txt audits are, by the Ahrefs data, the most active readers of the file anyway.
llms.txt is what a standard looks like before it is a standard: millions of publishers, almost zero consumers, and a two-year head start on whatever actually reads it. The v2 proposal's own framing concedes the point. Howard wrote that when the proposal first appeared in 2024, agents using websites was "largely a prediction. Today it is routine." Agents are reading the web in bulk now. They are just reading HTML, the same as always.
Sources
- llmstxt.org, "The /llms.txt file, v2" (Jeremy Howard, proposed September 3, 2024, updated August 10, 2026)
- Project LLMs.txt, "The real state of llms.txt adoption in 2026" (July 29, 2026)
- Ahrefs, "We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read" (June 2026)
- SE Ranking, "LLMs.txt: Why Brands Rely On It and Why It Doesn't Work"
- BuiltWith, llms.txt technology trends (checked October 17, 2026)
- Google Search Central, generative AI optimization guidance (updated May 15, 2026)
- Search Engine Journal, "Google Says llms.txt Is Purely Speculative For Now"
- Ethan Smith, "The Myth of LLMs.txt" (LinkedIn, sample of ~50,000 AI-cited URLs)
- Publii, "The Complete Guide to llms.txt" (updated October 18, 2026)
- Wix, "Agentic commerce on Wix"
- Chrome Lighthouse, llms.txt audit documentation
