We Crawled 1,629 Business Websites. Most Let AI In and Give It Nothing to Read.

Most of the public argument about AI and your website is about the gate. Should you let the crawlers in, or should you block them? It is the wrong argument, and we can now show why.

We crawled 1,629 business websites to find out what an AI system actually encounters when it arrives. Almost everyone lets it in. Almost nobody gives it anything to work with.

What we measured

Every site was checked on three layers.

Permission. Does robots.txt block GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot or the other AI crawlers?

Guidance. Does the site publish an llms.txt, the plain text file that tells an AI agent what the site is and which pages matter?

Structure. Does the site publish schema.org structured data, the machine-readable layer that states who the business is, who works there, what people have said about it, and what questions it answers?

Those three layers combine into a 0 to 10 AI-Readiness index, defined before the crawl ran and registered publicly so we could not move the goalposts afterward.

The sample covers 30 cities across five strata (metro, mid-size, small town, university, tourism) and 11 industries. We crawled 1,629 domains. 1,261 of them responded.

That gap is the first finding, and it arrives before any of the interesting ones. Twenty-three percent of these business websites did not respond at all. Nearly one business in four whose website is a matter of public record does not currently have a working one.

The headline: a 2 out of 10

Across the 1,261 reachable sites, the mean AI-Readiness score is 3.77 out of 10. The median is 4.

The number that matters is the mode. Four hundred ninety-nine sites, 40 percent of the sample, score exactly 2 out of 10. A 2 has a specific meaning in our rubric. It means the site permits AI crawlers and does nothing else. No machine-readable identity, no named people, no structured answers. The door is open and the rooms are empty.

What is actually on these sites

Adoption across the 1,261 reachable sites, with 95 percent confidence intervals:

Read the bolded row against the rest. Five percent of businesses block AI. That is the thing the industry argues about. Meanwhile 89 percent never name a single human being in a format a machine can read, and 96 percent publish no structured answers to any question.

The gate is not the problem. Everyone has already left the gate open. The problem is that there is nothing inside worth citing.

The llms.txt mirage

A quarter of sites have an llms.txt, which sounds like fast adoption for a standard barely a year old. It is not what it appears to be.

At least a quarter of those files carry the signature of a tool that generated them automatically, most often a WordPress SEO plugin such as Yoast or Rank Math. The remaining files carry no detected generator, and it is worth being precise about what that means. It means we found no tool signature. It does not mean a human wrote the file. When we classified provenance strictly in our study of 556 accounting firms, only about 9 percent of firms had genuinely authored their own.

Presence is not intent. A file your plugin emitted describes your platform. It does not describe your practice.

Some industries are further along than others

The gap between professions is real and it is wide.

Lawyers are the most structured industry we measured, and by a distance. On every one of the three signals they lead. Accounting sits at the bottom on identity, which is a striking place for a profession whose entire product is credentialed expertise.

Two industries in our sample, plumbing and HVAC, returned too few sites to report. Our protocol requires at least 40 sites in a cell before we publish a rate for it, and both came in under that. We are not going to print a percentage we cannot stand behind, so those cells are pooled and flagged rather than reported.

The most important result is the boring one

We have now run this crawl four times, at four different scales: 149 sites, then 487, then 766, and now 1,261.

The numbers barely move. Identity schema lands within a few points of 50 percent every time. Review schema stays near 8 to 11 percent. Person schema stays near 10 to 13. The share of sites blocking AI stays at 4 to 5. The share scoring exactly 2 out of 10 stays around 40.

That stability is the actual finding. A single crawl is a snapshot and can be dismissed as an artifact of who happened to be in the sample. Four crawls, at four scales, in different cities, converging on the same numbers, is a description of the landscape. It means the picture above is not a quirk of our sample. It is what the web looks like right now.

What to do about it

Stop worrying about the gate. You almost certainly are not blocking AI crawlers. Confirm it once, then move on. This is the least of your problems.

State who you are. Identity schema is the floor, and half of all businesses are still below it. It is a small, one-time technical change.

Name your humans. Person schema is the single biggest gap in the data. Eighty-nine percent of businesses have not told any machine that a specific, credentialed person works there and knows a specific subject. When an AI is asked to recommend an expert, it needs to know an expert exists. Most sites never say so.

Write your own llms.txt, or do not count the one you have. If a plugin generated it, it is describing your software, not your business.

Structure your answers. Four percent of sites publish FAQ schema. Answer engines are, definitionally, in the business of answers. Almost nobody is handing them one.

Method and data

The index rubric, the sampling frame and the analysis plan were registered before the confirmatory crawl ran, at OSF (DOI 10.17605/OSF.IO/2Q5ER). Proportions use Wilson score intervals. The crawler, the analysis code and the full dataset are public at github.com/Axion-Deep-Labs/ai-readiness-2026, the code under MIT and the data under CC BY 4.0. A revised working paper covering this confirmatory wave follows.

If you want to know what your own site looks like to an AI crawler, our free audit checks the same layers we measured here.