llms.txt is a plain Markdown file at the root of a website (/llms.txt) that gives AI tools a short summary of the site and links to its most useful pages. It’s a proposed convention, not a standard: it can’t block or rank anything, and search engines aren’t committed to using it. It is cheap to add, so treat it as a bonus, not a strategy.
We publish one on both of our websites, generated automatically every time the sites are built. This guide explains what the file is, shows what ours contains, clears up how it differs from robots.txt and sitemaps, and gives an honest view of what it does for you.
What is llms.txt?
llms.txt was proposed in September 2024 by Jeremy Howard of Answer.AI, and the proposal lives at llmstxt.org. The idea is simple. Web pages are built for people and browsers: navigation, scripts, cookie banners and layout wrap around the actual content. A large language model reading your site has to dig through all of that. A short, clean Markdown file that says “this is who we are, and these are the pages that matter” is much easier to use.
It is not a replacement for anything. Your pages, sitemap and robots.txt all keep doing their jobs. llms.txt is an extra summary layered on top.
What goes in an llms.txt file?
The proposal defines a simple structure:
- An H1 with the name of the site or company. This is the only required part.
- A blockquote with a one- or two-sentence summary.
- Optional detail in plain paragraphs: what you do, who for, key facts.
- H2 sections of links, each a Markdown list in the form
[Page name](URL): short note. - An optional section called “Optional”, for links a tool can skip when it has limited room.
Here is the start of our own file, trimmed for length:
# CyberSpark
> CyberSpark delivers bespoke software engineering, enterprise cloud platforms, and dedicated offshore development teams for UK companies. […]
CyberSpark is the trading name of Cyber Spark IT Services Ltd (Company No: 14891489), 71-75, Shelton Street, Covent Garden, London, United Kingdom, WC2H 9JQ. Contact: info@cyberspark.uk.
## Services
- [Bespoke Software Development Company UK](https://www.cyberspark.uk/services/bespoke-software-development): CyberSpark is a UK bespoke software development company. We design and build custom business software, web platforms and integrations that you fully own.
- [Web Development Company UK · Websites & Web Apps](https://www.cyberspark.uk/services/web-development): CyberSpark is a UK web development company building fast, accessible websites, online stores and web applications that you own, with support after launch.
## Products
- [AI Visibility Check by CyberSpark](https://www.cyberspark.uk/products/ai-visibility-check): Free AI visibility checker — see in 60 seconds whether Google and AI tools like ChatGPT recommend your business. Then let us fix it with local SEO and GEO.Three things we learned building it:
- Generate it, don’t hand-write it. Ours is built from each page’s own title and meta description, so it can’t drift out of date when pages change. A hand-edited file quietly goes stale.
- Your meta descriptions become your AI summary. Writing the file showed us which descriptions were vague or out of date. Improving them helped the file, the search results and the pages all at once.
- Include the facts people get wrong. Legal name, company number, location and contact details help any tool that is trying to work out who you are.
llms.txt vs robots.txt vs sitemap.xml: what’s the difference?
| File | What it does | Who it’s for | Can it block access? |
|---|---|---|---|
| robots.txt | Tells crawlers which paths they may or may not crawl | Search and AI crawlers | It asks; well-behaved crawlers comply |
| sitemap.xml | Lists every URL you want indexed, with update dates | Search engines | No |
| llms.txt | Summarises the site and highlights the most useful pages, in Markdown | AI tools and LLM-based agents | No |
The most common mistake is expecting llms.txt to control AI use of your content. It can’t. If you want to allow or refuse specific AI crawlers, that is robots.txt’s job, using the user-agent names each company publishes, such as OpenAI’s GPTBot and OAI-SearchBot, Anthropic’s ClaudeBot and PerplexityBot.
Do Google, ChatGPT and other AI tools use llms.txt?
This is where much of what’s written about llms.txt goes too far, so here is what is actually known:
- Google has said it does not use llms.txt for Search. Google’s AI Overviews are built on its normal search index, so they depend on your pages being crawlable and useful, not on this file.
- Chrome’s Lighthouse documentation now describes llms.txt in its section on agentic browsing, which suggests browser-based AI agents are one audience for it.
- Other AI companies have not published clear, consistent commitments to read llms.txt when answering questions. Some developer tools and documentation platforms do use it, mainly for technical docs.
So the honest answer is: some tools use it, the big search engines don’t say they do, and nobody can promise it changes how often you are mentioned in AI answers. Anyone selling llms.txt as a way to rank in ChatGPT is overselling it.
How do you create an llms.txt file?
For most business websites it takes minutes:
- List your most important pages: services, products, pricing or how-to-buy pages, about, contact and your best guides.
- Write one clear line for each. Your meta descriptions are a good start if they are specific.
- Add the facts: your full company name, what you do, where, and how to get in touch.
- Format it as Markdown in the structure above, with an H1, a blockquote, then H2 sections of links.
- Publish it at the root of your domain as plain text, and check it loads at /llms.txt.
- Keep it current. If your site runs on a framework with a build step, generate it automatically, as we do. On WordPress, a maintained plugin can do the same.
A generator can give you a first draft, but read the result. A summary of your business written by a tool, unchecked, is exactly the kind of vague text that doesn’t help anyone.
What matters more for AI search than llms.txt?
If you have an hour to spend on AI visibility, llms.txt should take five minutes of it. The rest matters more:
- Let the right crawlers in. Check that robots.txt, your firewall or your CDN aren’t blocking the AI crawlers you want to be found by.
- Put the answer on the page. Clear headings that match the questions people ask, with a direct answer underneath, are easy for any system to quote.
- Make your facts consistent. Your name, address, services and descriptions should match across your site, Google Business Profile, directories and social profiles.
- Use structured data. Organisation, service, product, FAQ and article markup help machines understand what a page is about.
- Be worth citing. Original, specific content, such as real examples, comparisons and clear explanations, gets referenced; generic content doesn’t.
You can see where you stand today with our free AI Visibility Check. If you want the work done properly, our AI search visibility and technical SEO services cover crawler access, structured data and content, and llms.txt too. Or get in touch with your questions.
Frequently asked questions
Is llms.txt required?
No. It is a proposed convention, not a web standard or a legal requirement. Nothing breaks if you don't have one. It is worth adding because it takes little effort, not because anything depends on it.
Can llms.txt stop AI from using my content?
No. llms.txt only describes and points to content; it has no power to allow or block anything. To control which crawlers can access your site, use robots.txt with the crawlers' published user-agent names, such as GPTBot, ClaudeBot or PerplexityBot, and remember that robots.txt is a request that well-behaved crawlers follow, not a lock.
Where do I put llms.txt?
At the root of your domain, so it loads at yourdomain.co.uk/llms.txt, served as plain text. The proposal also allows files at sub-paths, for example for a documentation section, but the root file is the one most tools look for.
What is llms-full.txt?
A companion file some sites publish with the full text of their key pages in one Markdown document, rather than just links. It is mostly used by documentation sites, where an AI coding tool benefits from having all the reference material in one place. Most business websites don't need it.
Will llms.txt improve my Google rankings?
No. Google has said it doesn't use llms.txt for Search, and ranking in Google depends on crawlable, useful pages. Treat llms.txt as a small extra for AI tools, not as an SEO tactic.
