Skip to content

llms.txt

llms.txt is a markdown file at the root of a site that points an AI reader at the pages worth reading. It is a proposed convention, not a standard, and no assistant has committed to reading it or to citing what it lists, so treat it as a signal you control rather than a directive anything obeys.

What we measured

19.8% of reachable sites had an llms.txt file of any origin, 95% Wilson confidence interval 16.5 to 23.7. Splitting by provenance, 10.6% carried a generator signature from a plugin or platform and 9.3% did not, 95% CI 7.0 to 12.3. That 9.3% is the closest available figure for deliberate adoption. This result did not support its pre-registered hypothesis, which predicted llms.txt adoption under 15%.

Source: State of CPA Firm Websites 2026, confirmatory crawl 2026-07-11 across 68 metropolitan areas., n=464.

Sample framing: 556 firms crawled, 464 reachable, and both percentages use the reachable denominator. United States accounting firms. The provenance split is inferred from generator signatures in the file, not from asking anyone.

How to re-derive it: Figures reproduce from the published dataset. Parent study pre-registration DOI 10.17605/OSF.IO/2Q5ER.

Why it matters

  • Writing one is cheap and reversible. That is the honest case for it: low cost, uncertain payoff, no downside if it turns out to matter.
  • The provenance split is the more useful finding. Half the files in the wild were installed rather than written, which means the median llms.txt lists whatever a plugin decided to list.
  • Writing it forces a question worth answering anyway. Which ten pages actually answer something? Most sites have never been asked.
  • The failure modes are silent. A file with bare URLs, or one served as HTML, looks fine in a browser and is useless to the thing it was written for.

The failure

The failure: a sitemap in a trench coat
# Example Accounting

https://example.com/
https://example.com/about
https://example.com/services
https://example.com/services/tax
https://example.com/services/audit
https://example.com/blog/2019/03/holiday-hours
https://example.com/blog/2019/04/we-have-moved
https://example.com/contact
https://example.com/privacy
https://example.com/terms

Two problems. The links are bare URLs rather than markdown links, so nothing has a title and the file communicates no more than a sitemap does. And it lists every page instead of the pages that answer something, including two posts from 2019 and the privacy policy.

The fix

The fix: markdown links, and only pages that answer something
# Example Accounting

> Tax and audit services for construction
> businesses in New Mexico and West Texas.
> Licensed since 2005.

## Services

- [Construction tax accounting](https://example.com/services/tax):
  Percentage of completion, multi-state filing, and
  job costing for contractors.
- [Financial statement audits](https://example.com/services/audit):
  Reviews and compilations for bonding requirements.

## About

- [Dana Whitfield, CPA](https://example.com/team/dana-whitfield):
  Tax practice lead, licensed in New Mexico and Texas.

## Contact

- [Contact and locations](https://example.com/contact):
  Las Cruces and El Paso offices, hours, and phone.

Markdown links with real titles and a sentence of context each. The list is short on purpose. A file that points at everything points at nothing, and the pages omitted here are omitted because they do not answer a question somebody would ask.

Verify the fix

Changing the code is not the same as fixing the problem. Confirm it.

  1. Fetch the file from the live domain rather than checking it exists in your repository. A file that is not served is not a file.
  2. Confirm the content type is text or markdown, not text/html. A framework serving your 404 page with a 200 status is the failure this catches.
  3. Confirm every entry is a markdown link with a descriptive title. Bare URLs in a list carry no information and will fail a Lighthouse check on the same grounds.
  4. Open every URL listed. A file that points at moved or deleted pages is actively misleading.
  5. Check whether a plugin is generating it. If a generator signature is in the file, something other than you decided what it says.
Check yours is actually served, and actually parseable
# 1. Is it there, and is it text?
curl -s -o /dev/null \
  -w "status=%{http_code} type=%{content_type}\n" \
  https://example.com/llms.txt

# 2. A soft 404 returns 200 and serves your HTML
# error page. This is the most common failure.
curl -s https://example.com/llms.txt | head -c 200

# 3. Bare URLs are the other one. Count lines that
# are a URL with no markdown link around them.
curl -s https://example.com/llms.txt | \
  grep -cE '^[[:space:]]*https?://'

Step 1 should report 200 and a text content type, not text/html. Step 3 should report 0. Anything above zero is a line an AI reader gets no title for, which defeats the point of the file.

Exceptions and misconceptions

No assistant has committed to reading it

It is a proposed convention with real adoption among publishers and no published commitment from any major assistant to honor it. Anyone telling you it gets you into an assistant is describing a hope, not a mechanism.

It is a signal, not a directive

Unlike robots.txt, which well-behaved crawlers obey, llms.txt asks for attention it cannot compel. Listing a page does not cause it to be read, and omitting one does not cause it to be ignored.

Our 9.3% is a floor, not a measured hand-authorship rate

The manual bucket means no generator signature was detected, not that a person was verified to have written the file. Some of that 9.3% will be plugins we could not fingerprint. Treat it as the best available estimate of deliberate adoption and not as a precise one.

It does not replace anything

It is not a substitute for a sitemap, for structured data, or for the content being in your HTML in the first place. A site whose answers only exist after JavaScript runs gets nothing from an llms.txt pointing at them.

Watch it

Primary sources

The measurement above comes from our study, State of AI-Readiness on Business Websites 2026.

Related lessons

Check your own site for this

DeepAudit AI renders your page in a real browser and reports the affected code, so you can see exactly where each finding came from. Free, no signup.

Run a free audit

Last reviewed 2026-09-18. Checks covered: llms.txt, llms-full.txt.