The number of domains publishing an llms.txt file jumped from 951 in July 2025 to 16,670 by September 2026 — roughly 17.5x growth in about 14 months. Yet not a single major AI platform, including ChatGPT, Google's AI Overviews or Anthropic's Claude, has confirmed that it reads or relies on the file. That is the central tension around a proposed standard that aims to give AI crawlers a curated map of a website's most important content.

What llms.txt is

The llms.txt file is a proposed standard designed to help large language models better understand and use content from websites. Instead of letting AI crawlers wander a site, publishers provide a curated Markdown-formatted list of their most important pages. Unlike robots.txt and sitemaps, it targets AI models specifically — systems that might cite the content when generating answers.

The spec addresses two real problems. First, most AI crawlers read only basic HTML, not JavaScript-rendered content, so a plain-text structured file gives them something they can digest quickly. Second, most sites carry far more content than a crawler can usefully process. If a bot burns its crawl budget on navigation menus and old blog posts, it may never reach the product pages a publisher actually wants cited.

The file itself is simple: a lowercase llms.txt in Markdown, with an H1 for the site name, an optional blockquote description, and H2 sections grouping bulleted links, each with a colon-separated description.

Confirmed signals versus speculation

Here is what separates confirmed fact from tool-signal speculation.

Confirmed: no AI platform says it uses the file. Google's John Mueller confirmed on Bluesky that Google does not rely on llms.txt. In first-party testing on Search Engine Land, which implemented the file in March 2025, there was no correlation between publishing llms.txt and improved performance in AI results.

The server logs are even more blunt. From mid-August to late October 2025, the site's llms.txt page received zero visits from Google-Extended, GPTBot, PerplexityBot or ClaudeBot. Traditional crawlers like Googlebot and Bingbot did hit the file, but only a few times — no special treatment.

Speculative but notable: tooling is moving anyway. Google added an experimental Agentic Browsing category to Lighthouse, Chrome's site-auditing tool, and one check looks for an llms.txt file. That signals the format is being taken seriously, but Lighthouse audits and live crawling are separate systems, so it is not evidence that Google's crawlers use the file to answer queries.

Two other signals complicate the picture. Anthropic has published an llms.txt file on its own website — which does not mean its crawler reads these files, but suggests openness to the idea. And Google released the Open Knowledge Format (OKF), currently at version 0.1. OKF is not a replacement: llms.txt faces outward toward crawlers visiting a site, while OKF packages content for an organization's own internal AI agents.

How brands are using it

Adoption patterns vary widely. Hugging Face's file reads like a comprehensive knowledge base, with multiple heading levels, full code examples and annotations. Vercel opens with metadata-style lines (title:, description:, tags:) before organized sections with step-by-step instructions. Zapier keeps it to a long list of links with brief descriptions. Cal.com groups links under simple headings. All are valid — any well-formed Markdown is machine-readable.

Who should implement, and how

For most site types, llms.txt remains a low-cost experiment rather than a requirement. Content can still appear in AI answers without one. The barrier to entry is minimal: pick the pages that matter — product and service pages, current blog posts, pricing, about and contact — favoring evergreen, canonical content over time-sensitive or login-gated pages. Write the file in any text editor, then upload it to the root directory (or a docs subdomain) via the hosting control panel. Free generators and validators can speed up drafting, and Semrush Site Audit can confirm the file is picked up.

Publishers who deploy one should check server logs periodically for AI crawler requests and keep the file updated as pages change. For sites dependent on AI-driven referral traffic — documentation-heavy SaaS, developer tools, knowledge bases — the cost of being early is near zero. For everyone else, the file is a hedge, not a lever.

What to watch next

The honest framing is that llms.txt sits in an early speculation phase: publishers are implementing the file and hoping it becomes useful. The monitorable signals are concrete. Watch whether OpenAI, Google or Anthropic ever confirms crawler support in documentation; watch whether GPTBot, ClaudeBot or Google-Extended starts requesting the file in your own server logs; and watch the evolution of Google's OKF specification and the Lighthouse Agentic Browsing checks, which indicate where Google thinks AI content accessibility is heading — even if none of it, today, moves rankings or AI answer citations.