llms.txt Example: What to Put in It and Why

In short

llms.txt is a curated Markdown index at the root of your site — /llms.txt — that points language models at your best pages. The spec requires exactly one thing: an H1 with the project name. Convention adds a blockquote summary, then H2 sections holding lists of [name](url): short note links, plus an ## Optional section whose links can be dropped when context is tight. It is not a ranking signal and no search engine has committed to reading it. Google's own guidance says such files "will neither harm nor help your site's visibility or rankings in Google Search." Ahrefs looked at 137,210 domains in May 2026 and found 97% of published llms.txt files received zero requests. Where the format does earn its keep is agent-facing: a documentation index an LLM can be told to fetch, cheaply, instead of crawling HTML. Write it for that job, keep it under about 10 KB, and don't expect traffic from it.

What llms.txt is, and who actually reads it

The proposal is simple. HTML pages are mostly navigation, scripts and boilerplate, so a model reading them spends context on markup. Publish one Markdown file at /llms.txt listing your important pages with one-line descriptions, and let a model start there.

The spec defines a strict order:

1. An H1 with the name of the project or site — "this is the only required section."
2. A blockquote with a short summary "containing key information necessary for understanding the rest of the file."
3. Zero or more Markdown sections of any type except headings.
4. Zero or more H2-delimited sections, each containing a list of links in the form [name](url) optionally followed by : and notes.

The ## Optional H2 is special: URLs there "can be skipped if a shorter context is needed."

Now the part vendors gloss over: nothing crawls this file on its own. The Ahrefs study found AI bots never requested llms.txt on domains where it didn't exist — crawlers aren't probing for it. Of the small share of files that did get requests, the largest slice came from SEO audit tools (21.7%); AI retrieval bots accounted for 1.1%.

Where it genuinely works is different: a model or agent is handed the URL. Anthropic does this explicitly — fetch any page under code.claude.com/docs/ and the response opens with a pointer to the documentation index and the instruction to "use this file to discover all available pages before exploring further." That is llms.txt doing real work, because something told the agent it exists.

A working example, annotated

A compact file that follows the spec. Anything longer is a sitemap in disguise.

# Example Docs

> Example is an open-source job queue for Postgres. This index covers setup,
> the API reference and migration guides. Everything here is current for v4.

Prices and limits change; the pricing page is authoritative.

## Getting started
- [Install](https://example.com/docs/install): requirements, Docker and bare-metal setup
- [Quickstart](https://example.com/docs/quickstart): first queue and worker in 10 minutes
- [Configuration](https://example.com/docs/config): every env var with defaults

## Reference
- [Job API](https://example.com/docs/api/jobs): enqueue, retry, cancel, batch semantics
- [Worker API](https://example.com/docs/api/workers): concurrency, backoff, graceful shutdown
- [Error codes](https://example.com/docs/errors): what each code means and how to recover

## Guides
- [Migrating from v3](https://example.com/docs/migrate-v3): breaking changes and a rollback plan
- [Deploying on Kubernetes](https://example.com/docs/k8s): manifests and health probes

## Optional
- [Changelog](https://example.com/changelog): release notes since v1
- [Blog](https://example.com/blog): design write-ups, not needed to use the library

What makes this work rather than just exist:

• The blockquote carries the version and scope. A model reading only the first three lines already knows what it's dealing with.
• The notes answer "what does this page tell me?" — not "what is this page called". [Error codes]: what each code means and how to recover routes the model; [Error codes]: error codes is noise.
• Absolute URLs. Relative paths work in some parsers and break in others.
• ## Optional is used properly — the changelog and blog are genuinely droppable.
• No headings deeper than H2, no tables, no images, no code blocks. The structure is flat by design.

Serve it as text/plain or text/markdown at the exact path /llms.txt. Our own file is 11 KB across 62 lines, which already feels near the ceiling of what's useful.

llms.txt vs llms-full.txt vs robots.txt

These four files get conflated constantly, but they do unrelated jobs.

The llms-full.txt row deserves emphasis, because it is the one people get wrong. The spec never defines it; llmstxt.org describes llms-ctx.txt and llms-ctx-full.txt produced by the llms_txt2ctx tool instead. llms-full.txt became a de facto name because documentation platforms started emitting it, and it is usually enormous: ours is roughly 1.9 MB, about 175 times the size of our llms.txt and far past what any agent reads in one go. If your docs platform generates it for free, fine. Hand-maintaining one is wasted effort.

If your actual goal is controlling whether AI crawlers may use your content, robots.txt is the file that does that — see our card on answer engines and AI crawlers.

Does it change anything measurable?

Short answer: no evidence that it does, and reasonable evidence that it mostly doesn't.

Google is explicit. The Search Central AI optimization guide (updated 10 July 2026) states: "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them," and adds that they "will neither harm nor help your site's visibility or rankings."

The field data agrees. Ahrefs analysed 137,210 domains with traffic in May 2026 and published the results on 15 June 2026: 28% of them publish an llms.txt, and 97% of those files received zero requests that month. Of the ~1,100 domains that saw any traffic to the file, 96% of requests were bots — dominated by SEO audit tools, generic crawlers and tech-profiling services rather than AI systems.

Our own numbers say the same. We publish /llms.txt and /llms-full.txt here and have not been able to attribute a single referral, citation or visibility change to either. That isn't proof it does nothing — at our traffic level the effect is below the noise floor. But anyone claiming they measured a lift owes you their methodology.

So why publish one at all? The cost is close to zero and the payoff isn't a ranking — it's a working index for the agent case above. If a developer asks Claude Code or Cursor to "read the docs for X", a good llms.txt turns that into one fetch instead of ten. Real, unglamorous, and orthogonal to search. Our knowledge base and GEO/AEO card go deeper on what does and doesn't move AI visibility.

Writing one that isn't a waste of time

A short checklist, in the order that matters:

1. Pick 10–30 pages, not 300. Pillar guides, the API reference, the pricing page, the "how do I do the main thing" tutorial. If a page wouldn't make your own top-30 list, it belongs in the sitemap.
2. Write the blockquote last. One or two sentences on what the site is for, plus anything needed to read the rest: version, scope, language.
3. Make every note a promise about content. Twelve words maximum.
4. Push secondary material into ## Optional. Changelogs, blogs, legal pages.
5. Regenerate it when your docs change. A stale index is worse than none — it sends agents to renamed URLs.
6. Don't build Markdown mirrors of every page just for bots. Nobody has shown that pays.

If you maintain docs with a static-site generator, check for a plugin before writing anything by hand — llmstxt.org lists implementations for VitePress, Docusaurus and Drupal, plus JavaScript and PHP libraries.

FAQ

What is an llms.txt file?

It's a Markdown file at the root of a website — example.com/llms.txt — that lists a site's most important pages with short descriptions, so a language model can find the right page without parsing HTML. It's a curated index, not a complete inventory. The only structurally required element is an H1 with the site or project name.

Does llms.txt help SEO?

No. Google's Search Central documentation says it doesn't use these files and that they will neither help nor harm rankings. Ahrefs' May 2026 data found 97% of llms.txt files got no requests at all. Treat it as a convenience for agents that are pointed at it, not as a search tactic.

What's the difference between llms.txt and llms-full.txt?

llms.txt is a short curated index of links; llms-full.txt is the full text of your documentation concatenated into one file. Only the first is described by the spec at llmstxt.org — llms-full.txt is a convention popularised by documentation platforms. Full files run to megabytes, so publish one only if your tooling generates it automatically.

Where do I put llms.txt and what content type should it use?

At the site root, at exactly /llms.txt, served over HTTPS with a 200 response. Use text/plain; charset=utf-8 or text/markdown. Don't gate it behind redirects, bot challenges or a CDN rule that returns HTML — the most common failure is a file that returns your 404 page as plain text.

Do I need an llms.txt generator?

Not for a small site — hand-writing 20 curated links takes twenty minutes and produces better notes than any generator. For large documentation sets, use a plugin for your static-site generator so the file regenerates with your docs. The risk with generators is that they dump every URL they can find, which defeats the point of a curated index.