What goes in an llms.txt
The format is small on purpose. One heading with the name of the site, one line that says what it is, then sections of links with a short note on each. Secondary pages, like the blog archive or legal pages, go under a section called Optional, which a model can skip when it is short of space.
| File | Written for | What it says |
|---|---|---|
| robots.txt | All crawlers | What they may and may not fetch |
| sitemap.xml | Search engines | Every page, for discovery |
| llms.txt | AI models and agents | What the site is and which pages to read first |
How the generator builds your file
It starts from your sitemaps, following sitemap index files, or from your homepage links when there is no sitemap. It leaves out pages that carry nothing for a model: login and account pages, carts and checkout, tag archives, pagination, feeds, and anything marked noindex. From what is left it keeps your homepage and your most important pages, spread across the sections of the site so one large blog cannot crowd out everything else.
Each page's title and description come from the page itself, with the site name trimmed from titles, since it is already the file's heading. Pages that share a section of your site, like /docs or /products, are grouped under that heading. The file is then checked against the format before you see it.
PublishingPublishing it
Save the file as llms.txt and serve it from the root of your domain as plain text, so it opens at yourdomain.com/llms.txt. Then run the checker on the live address to confirm it is found and its links open. Keep it short and edit it by hand: you know better than any generator which pages should be first.
What llms.txt will not do
It will not make an AI engine cite you, and it will not unblock a crawler that robots.txt or your firewall keeps out. For that, run the AI crawler checker. To see which AI answers cite your site today, run the free AI visibility audit.