Does Your Site Need an llms.txt File? What the 2026 Data Shows

Short answer: probably not, and certainly not for the reason it is usually sold to you. An llms.txt file takes about twenty minutes to make. The problem is not the cost. The problem is that almost nothing reads it, and the people telling you it will get your pages cited in AI answers are describing something the data does not support.

Here is what the file actually is, what the evidence says about whether it works, and where that twenty minutes goes further.

What an llms.txt file is

An llms.txt file is a plain Markdown file that sits at the root of your domain, at yourdomain.com/llms.txt, in the same place as robots.txt. Inside it you write a short description of your site and a curated list of links to the pages that matter most, each with a sentence explaining what it covers.

Jeremy Howard, co-founder of Answer.AI and fast.ai, proposed the idea in September 2024. The reasoning was sound. A web page is full of navigation, scripts, cookie banners, and footers, and a language model working inside a limited context window has to wade through all of it to reach the part that matters. A clean Markdown index that says “read these pages, in this order” solves a genuine problem, at least on paper.

It is worth being precise about one thing, because it is often confused. llms.txt is not robots.txt. Robots.txt restricts what a crawler may access. An llms.txt file does the opposite — it recommends your best content. They are not alternatives, and having one says nothing about the other.

Do AI crawlers actually read it?

This is the question that matters, and the honest answer is no.

The most substantial work on this is SE Ranking’s analysis of roughly 300,000 domains, published in November 2025, which set out to test whether having the file correlates with being cited more often by AI systems. It does not.

Start with adoption. They found llms.txt on 10.13% of the domains they looked at — nearly nine sites in ten have not implemented it. That alone is a long way from the near-universal adoption of robots.txt or XML sitemaps.

The more interesting number is how that adoption breaks down by site size. Low-traffic sites adopt at 9.88%, mid-traffic sites at 10.54%, and high-traffic sites at 8.27%. The largest, most authoritative domains are slightly less likely to have the file than mid-tier ones. If llms.txt were quietly driving AI visibility, you would expect the sites winning at AI visibility to have noticed. They have not.

Then the direct test. SE Ranking ran both conventional statistical analysis and a machine learning model to look for a relationship between having the file and how often a domain gets cited in AI answers. They found none. When they removed llms.txt from the model entirely, the model’s prediction accuracy improved — meaning the file was contributing noise rather than signal. Search Engine Journal covered the findings when they were published.

Google’s own position is documented rather than merely reported. Its guidance on AI features and your website states plainly that there are no additional requirements to appear in AI Overviews or AI Mode, and no special optimizations necessary — the same SEO fundamentals apply. OpenAI, for its part, points site owners to ordinary robots.txt crawler permissions rather than to any bespoke file.

It is also worth knowing that llms.txt is not a standard in any formal sense. It is a community proposal with no backing from the W3C, the IETF, or any recognized standards body, and no enforcement mechanism. Anyone describing it as a web standard is overstating what it is.

One caveat, stated plainly: that study is from late 2025, and this is a field where a year is a long time. It remains the largest analysis anyone has published, and nothing since has contradicted it — but it is evidence with a date on it, not a permanent law. If a major provider announces support tomorrow, this article is out of date tomorrow.

So why do so many articles recommend it?

Partly because it is easy to recommend. The file is cheap to produce, it sounds technical, and “add this file to your site” is a satisfying piece of advice to give and to receive. It has the shape of a shortcut.

Partly because a plugin will generate one for you in a single click, which means a large share of the files in the wild are empty stubs that nobody wrote and nobody maintains.

And partly because AI search is new enough that confident advice travels faster than evidence. This is worth holding onto beyond this one file. When a tactic is described as essential for AI visibility, the useful question is always the same: has anyone measured it, or does it just sound right?

When it is actually worth having one

There are narrow cases where the file earns its place.

• Documentation sites. Coding agents are the one category of consumer that demonstrably reads these files today. If you publish developer documentation, this is a real use case rather than a speculative one.
• Internal pipelines. If you point your own tools at a curated index of your content, the file is a convenient place to keep it.
• A long-term bet you can afford. Twenty minutes is twenty minutes. If you want the file in place in case adoption changes, that is a defensible reason, as long as you are honest with yourself that you are buying an option rather than a result.

What does not belong on that list is any expectation of more citations, better rankings, or improved AI visibility in the near term. There is currently no verifiable evidence for it.

Where that time goes further

If you have a limited number of hours to spend on being understood by AI search, spend them here instead, roughly in this order.

Make sure crawlers can reach you at all. Check that your robots.txt is not blocking the AI user agents you want reading your site, and that your pages render their content in HTML rather than assembling it with JavaScript after load. Access is the prerequisite for everything else.

Answer the question in the first two sentences. An answer engine quoting your page needs a self-contained passage it can lift. A paragraph that opens with three sentences of preamble before reaching the point is much harder to quote than one that states the answer and then explains it.

Structure the page so passages stand alone. Descriptive headings, short paragraphs, and sections that make sense out of context. This is the same discipline that makes a page easy for a person skimming on a phone.

Publish something only you can publish. Original numbers, a method you actually use, a result you measured. Aggregated advice gets summarized without attribution because there is nothing in it worth attributing. A specific claim that only exists on your site gets cited because it has to be.

We cover this in more depth in AEO and GEO Explained: How to Make Your Content Easier for AI Search to Use.

How to set one up, if you still want to

Create a plain text file named llms.txt and upload it to the root of your site so it resolves at yourdomain.com/llms.txt. The format is Markdown, and the convention looks like this:

# Your Site Name

> One sentence describing what the site is and who it is for.

## Core pages

- [Page title](https://yourdomain.com/page/): One sentence on what this page covers.
- [Another page](https://yourdomain.com/other/): One sentence on what this page covers.

## Background

- [Supporting article](https://yourdomain.com/article/): One sentence on what this page covers.

Two things to avoid. Do not let a plugin generate a Markdown copy of every page on your site — if those copies are indexable, you have created duplicate content at scale, which is a real cost against a speculative benefit. And do not treat the file as access control. A crawler that ignores it loses nothing, because there is nothing to enforce.

If you want to know whether anything is reading yours, filter your server access logs for requests to /llms.txt by known AI user agents, or put a unique URL inside the file that appears nowhere else and watch for hits on it.

Frequently asked questions

Is llms.txt an official web standard?

No. It is a community proposal with no backing from the W3C, the IETF, or any recognized standards body, and no enforcement mechanism. AI providers can adopt it or ignore it entirely.

Is llms.txt the same as robots.txt?

No, and they do close to opposite jobs. Robots.txt restricts crawler access. An llms.txt file recommends your best content to language models. Keep your robots.txt regardless of what you decide here.

Will an llms.txt file help me rank in Google?

No. Google’s own documentation states there are no additional requirements or special optimizations needed to appear in AI Overviews or AI Mode — the ordinary SEO fundamentals are what apply.

Does ChatGPT read llms.txt?

There is no confirmation that it does. GPTBot has been observed fetching the file occasionally, but a fetch is not evidence that the contents influence how ChatGPT sources, ranks, or cites anything.

Will this change?

It might. The main evidence here dates from November 2025; adoption could shift, and the agentic side of the web is moving quickly. The file is cheap enough that watching this space costs you nothing. Just do not spend your limited hours on it ahead of crawler access, answer structure, and original content.

The short version

An llms.txt file is a reasonable idea that the ecosystem has not adopted. If you want one, twenty minutes is a fair price for an option on the future. If you are shipping one because you were told it would get you cited in AI answers, you were told something that the evidence does not currently support — and the hours would do more work on the four things above.