Robots.txt
SEO/AEO/GEOA small file at the root of your site that tells search engine crawlers which pages they can and can't access — useful for keeping junk pages and crawler traps…
AI & Search
LLMs.txt
llms.txt file
Lukas HorvathCo-founder
What is LLMs.txt?
An llms.txt file is a markdown file placed at the root of a domain (yoursite.com/llms.txt) that gives AI models a curated map of the site's most important content. It lists key pages, summaries, and links in a format LLMs can parse quickly. Unlike robots.txt, which controls crawling, llms.txt helps models find and understand the content you want them to cite.
Why it matters
Most websites are built for human readers and Googlebot. AI models have to guess which pages are canonical, which are marketing fluff, and which are out of date. An llms.txt file removes the guessing. You tell Claude, ChatGPT, and Perplexity exactly what your product does, where the docs live, and which case studies are real. The teams that publish a good llms.txt today get pulled into AI answers more often. The teams that ignore it let the model improvise — and improvised answers are how competitor names end up in your buyer's chat window.
How it works
You create a markdown file at yoursite.com/llms.txt with a short site description, then sectioned links to key pages — product, docs, pricing, customer stories — each with a one-line summary. Some teams also publish llms-full.txt, which inlines the actual content of those pages so models can ingest it without crawling. The standard was proposed by Jeremy Howard in 2024 and adopted quickly by tool vendors, search engines, and AI labs. You write it once, update it when your site changes meaningfully, and treat it like a sitemap for AI. No CMS plugin required — it is just a text file you ship alongside the rest of your static assets.
Related terms
AI Crawler
AI & SearchAn automated bot that AI companies use to read websites and feed the content into their models or live answer engines — including GPTBot, ClaudeBot,…
Structured Data for LLMs
AI & SearchMarking up your website content with schema, clean HTML, and machine-readable structure so AI models can extract and cite it accurately — the technical…
Answer Engine Optimization
SEO/AEO/GEOOptimizing your content so AI answer engines like ChatGPT, Perplexity, and Google AI Overviews quote you directly when buyers in your market ask a question —…
Generative Engine Optimization
SEO/AEO/GEOShaping your content so generative AI tools like ChatGPT and Gemini surface your brand inside their answers — the next layer of search, built around language…
AI Citation
AI & SearchWhen an AI model credits your website as a source in its answer — the new equivalent of ranking on page one, and increasingly the most important metric to…
Brand Mentions in AI
AI & SearchHow often and in what context your company gets named when people ask AI models questions in your category — the new equivalent of share of voice in organic…