Axion Deep Digital Research / July 2026
We crawled 1,629 business websites. Most let AI in and give it nothing to read.
The industry argues about whether to block AI crawlers. Almost nobody blocks them. The real finding is what the crawler finds when it arrives: an open door and an empty room.
The numbers that define this report
score exactly 2 out of 10 on the AI-Readiness index. A 2 means the site permits AI crawlers and does nothing else. 499 of 1,261 reachable sites.
never name a single human being in a format a machine can read. No Person schema, so no expert exists as far as an answer engine is concerned.
block any AI crawler. This is the thing the industry argues about, and it is the rarest signal on the web.
have an llms.txt, which looks like fast adoption of the new AI-agent standard.
of the sites never answered. Nearly one business in four whose website is on public record does not have a working one.
Pre-registered. Open data. Open code.
1,629-row dataset. The rubric and analysis plan were registered before the crawl ran.
What we measured
Every site was checked on three layers. Permission: does robots.txt block GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot or the others? Guidance: does the site publish an llms.txt, the file that tells an AI agent what the site is and which pages matter? Structure: does the site publish schema.org data stating who the business is, who works there, and what questions it answers?
Those three layers combine into a 0 to 10 AI-Readiness index. The rubric, the sampling frame and the analysis plan were registered publicly at OSF before the confirmatory crawl ran, so we could not move the goalposts after seeing the data. Every figure is descriptive. We make no causal claims about what any AI system does with these signals.
The headline: a 2 out of 10
The mean score is 3.77 out of 10 and the median is 4. The number that matters is the mode: 499 sites, 40 percent of the sample, score exactly 2 out of 10. In our rubric a 2 has one meaning. The site permits AI crawlers and does nothing else.
The distribution is worth a second look. Sites do not spread evenly across the scale. They pile up at 2, then at 4, then at 6, with almost nobody at 1, 3, 5 or above 8. That is not what a gradient of effort looks like. It is what a platform default looks like. Businesses are not choosing a level of AI readiness. They are inheriting whichever one their website builder happened to ship.
What is actually on these sites
| Signal | Adoption | 95% CI |
|---|---|---|
| Identity schema (Organization / LocalBusiness) | 53% | 50–56 |
| llms.txt present (any provenance) | 25% | 23–28 |
| Person schema (a named human expert) | 11% | 9–12 |
| Review / rating schema | 8% | 7–10 |
| Blocks at least one AI crawler | 5% | 4–7 |
| FAQ / Q&A schema | 4% | 3–5 |
Read the blocking row against the rest. Five percent of businesses block AI. Meanwhile 89 percent never name a human being in a format a machine can read, and 96 percent publish no structured answer to any question. The gate is not the problem. Everyone has already left it open. The problem is that there is nothing inside worth citing.
The llms.txt mirage
A quarter of sites have an llms.txt, which sounds like fast uptake for a standard barely a year old. At least a quarter of those files carry the signature of a tool that generated them automatically, usually a WordPress SEO plugin such as Yoast or Rank Math.
The rest carry no detected generator, and it is worth being precise about what that means. It means we found no tool signature. It does not mean a human wrote the file. When we classified provenance strictly in our study of 556 accounting firms, only about 9 percent of firms had genuinely authored their own. Presence is not intent. A file your plugin emitted describes your platform, not your practice.
Some industries are further along than others
| Industry | Sites | Identity | Review | Person |
|---|---|---|---|---|
| Lawyers | 177 | 64% | 22% | 28% |
| Dentists | 240 | 62% | 9% | 12% |
| Veterinary | 100 | 53% | 2% | 6% |
| Real estate | 132 | 51% | 3% | 6% |
| Beauty & med spa | 243 | 50% | 6% | 5% |
| Opticians | 73 | 48% | 1% | 3% |
| Auto repair | 209 | 43% | 8% | 8% |
| Accounting | 51 | 35% | 6% | 10% |
Lawyers are the most structured industry we measured, and by a distance. They lead on all three signals. Accounting sits at the bottom on identity, which is a striking place for a profession whose entire product is credentialed expertise.
Two industries in the sample, plumbing and HVAC, returned fewer than 40 sites. Our protocol sets a 40-site floor before we publish a rate, so they are pooled out and not reported. We have their numbers. They are not trustworthy at that sample size, and a percentage printed from 16 sites would look exactly as authoritative as one printed from 240.
The most important result is the boring one
We have now run this crawl four times, at four different scales. The numbers barely move.
| Wave | Reachable sites | Scoring 2/10 | Identity | Person |
|---|---|---|---|---|
| Pilot | 149 | ~50% | 35% | 11% |
| Wave 2 | 375 | 42% | 52% | 13% |
| Wave 3 | 594 | 42% | 49% | 10% |
| Confirmatory | 1261 | 40% | 53% | 11% |
That stability is the actual finding. A single crawl is a snapshot and can be dismissed as an artifact of whoever happened to land in the sample. Four crawls, at four scales, across different cities, converging on the same numbers, is a description of the landscape. The picture above is not a quirk of our sample. It is what the web looks like right now.
What to do about it
Stop worrying about the gate. You almost certainly are not blocking AI crawlers. Confirm it once, then move on.
State who you are. Identity schema is the floor and half of all businesses are still below it. It is a small, one-time technical change.
Name your humans. Person schema is the single biggest gap in the data. When an AI is asked to recommend an expert, it needs to know that an expert exists. Most sites never say so.
Write your own llms.txt, or do not count the one you have. If a plugin generated it, it is describing your software.
Structure your answers. Four percent of sites publish FAQ schema. Answer engines are, by definition, in the business of answers. Almost nobody is handing them one.
Methodology
Sample. 1,629 business domains drawn from OpenStreetMap points of interest carrying a website tag, stratified across 30 US cities in five metro strata (metro, mid-size, small, university, tourism) and 11 industries. 1,261 responded and form the denominator for every adoption figure reported here. Unreachable sites are excluded rather than counted as having no schema, because a dead URL is not a markup failure.
Measurement. For each site we fetched the homepage, robots.txt and llms.txt, extracted all JSON-LD, and recorded which signals were present. The AI-Readiness index awards 2 points each for allowing AI crawlers, publishing an llms.txt, publishing identity schema and publishing review schema, plus 1 point each for FAQ and Person schema.
Statistics. Every proportion carries a Wilson score 95% confidence interval, which behaves correctly near 0% and 100% where a percentile bootstrap collapses to a zero-width interval. The index mean uses a bootstrap. Cells below 40 sites are pooled and flagged rather than reported.
Limits. This measures structure, not AI behavior. We did not ask any AI system what it recommends, so nothing here establishes that these signals cause an AI to cite a business. The index is an operational benchmark, not a validated construct. Homepage-only measurement will undercount sites that publish schema exclusively on interior pages.
Open materials. The crawler, the analysis code and the full domain-level dataset are public at github.com/Axion-Deep-Labs/ai-readiness-2026, code under MIT and data under CC BY 4.0. The pre-registration is at osf.io/2q5er (DOI 10.17605/OSF.IO/2Q5ER). A revised working paper covering this confirmatory wave follows.
What does your site look like to an AI crawler?
Our free audit checks the same layers we measured here: crawler permission, structured data, and whether a machine can tell who you are and what you know.
Run a free audit