llms.txt

Written by Yatin Malik, Founder · Updated May 2026 · 7 min read

llms.txt is a plain-text file served at the root of a website — yoursite.com/llms.txt — that gives large language models a machine-readable summary of what the site is, what matters on it, and which pages contain the canonical version of each topic. Where robots.txt controls *access*, llms.txt provides *context*: it tells AI crawlers like GPTBot, ClaudeBot, and PerplexityBot how to understand your site instead of just whether they can read it.

What llms.txt contains

The format was proposed by Jeremy Howard of Answer.AI in September 2024 and adopted quickly by AI-first sites because it solved a real problem: HTML pages bury the key facts behind navigation, scripts, and design. A markdown-formatted llms.txt strips that away and gives a model a clean, hierarchical reading list of the most important pages on the site.

A well-formed file starts with an H1 of the site name, a single-sentence description, and then a short paragraph of context. After that comes one or more H2 sections — typically "Docs", "Examples", "Optional" — each containing a markdown-style bulleted list of URLs, each with a short description. The spec is deliberately minimal: machine-readable, human-editable, no required tooling. The companionllms-full.txt variant can inline the actual content of the linked pages for models that want a single fetch.

Major AI-tooling vendors (Anthropic, Mintlify, Cloudflare, Vercel) shipped llms.txt support within months of the proposal. Adoption among non-AI brands is still uneven, which means publishing one today is a low-cost differentiator: most of your competitors don't have one yet.

Why llms.txt matters for AI visibility

AI crawlers have limited budget per site. If a model spends half its visit parsing your marketing JavaScript, it never reaches the comparison page that would have answered the buyer's question. A pointed llms.txt directs the crawler to the pages that matter — your definitions, comparisons, methodology, pricing — and skips the visual chrome. This compresses what the model learns about you per crawl and improves the odds it cites the page you want it to cite.

llms.txt is also one of the simplest signals you can send. There is no Schema.org vocabulary to learn, no JSON-LD validator to satisfy, no deploy pipeline gymnastics. A junior marketer can write a passable llms.txt in 20 minutes. Doing it is closer to a tax than an investment — but it's a tax that pays a small return on every AI crawl from now until the standard becomes irrelevant.

How to create and deploy llms.txt

You can hand-write one in any text editor — there's no required tool. Or use our free llms.txt generator, which takes your domain and produces a valid file in 30 seconds. Either way, place the finished file at the root path (not in a subdirectory). Serve it withContent-Type: text/plain and a permissive cache header.

Update it whenever your site structure changes meaningfully — new product line, new pricing page, new pillar content. Treat it like the sitemap's editorial cousin: not the machine-generated full index, but a curated "these are the pages we want AI to know about" pointer. AI Autopilot ships llms.txt automatically as part of its deployment bundle and keeps it in sync with the rest of your structural signals.

Frequently Asked Questions

What is llms.txt?

llms.txt is a plain text file placed at the root of a website (yoursite.com/llms.txt) that provides a machine-readable summary of the site's content, structure, and key information. It is designed to help AI crawlers and large language models understand your site more effectively.

How is llms.txt different from robots.txt?

robots.txt tells crawlers which pages they can or cannot access. llms.txt is complementary; it provides a structured summary of your site's content to help AI models understand what your site is about and what information it contains, improving the chances of accurate citation.

Do I need a llms.txt file?

While not yet universally adopted, adding a llms.txt file is a low-effort, high-potential GEO tactic. It helps AI crawlers like GPTBot and PerplexityBot quickly understand your site content, which can improve your chances of being cited in AI-generated responses.