Blocking AI Crawlers

Blocking an AI crawler takes a deliberate robots.txt rule, and almost nobody has written one. In a crawl of 464 accounting firm websites, 2.4% blocked any AI crawler at all, so if an assistant never mentions a business the cause is usually what the crawler found rather than whether it was let in.

What we measured

2.4% of reachable sites blocked at least one AI crawler, 95% Wilson confidence interval 1.3 to 4.2. This was one of four pre-registered hypotheses and the result supported it: AI blocking is rare. The broader business-website crawl put the same measure at 5%, 95% CI 4 to 7, over a reachable denominator of 1,261. Both say the same thing. The door is open almost everywhere.

Source: State of CPA Firm Websites 2026, confirmatory crawl 2026-07-11 across 68 metros, same crawler as the AI-Readiness study., n=464.

Sample framing: 556 firms crawled, 464 reachable, and every statistic here uses the reachable denominator. United States accounting firms across 68 metropolitan areas. It measures structured-data legibility and crawler policy, not firm quality.

How to re-derive it: Pre-registration DOI 10.17605/OSF.IO/2Q5ER for the parent study. Figures reproduce from the published dataset.

Why it matters

It is the first thing everyone checks and the least likely thing to be wrong, which makes it an expensive place to spend an afternoon.
Blocking is a decision somebody makes. Invisibility is a default nobody chose. Only one of those is fixed in robots.txt.
Knowing the real rate changes what you look at next. At 2.4%, the base rate says look at the page, not the policy.
There are legitimate reasons to block, and confusing an intentional block with an accident wastes the decision somebody already made deliberately.

The failure

The rare real block, and the far more common false alarm

# An actual block. This is what 2.4% of sites have.
User-agent: GPTBot
Disallow: /

# Not a block. This is a crawl-rate request and is
# routinely misread as one.
User-agent: *
Crawl-delay: 10

# Also not a block on AI crawlers. This only
# affects Google's use of content for Gemini,
# and leaves Google Search indexing untouched.
User-agent: Google-Extended
Disallow: /

Only the first rule stops GPTBot. Google-Extended is a separate opt-out that does not affect Google Search, and a crawl delay is not a disallow. Reading these three as the same thing is how sites get diagnosed as blocked when they are not.

The fix

An explicit policy, so the answer is not left to a default

# Decide per crawler and write it down.
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Keep the crawlers out of what no reader needs.
User-agent: *
Disallow: /cart/
Disallow: /account/

Sitemap: https://example.com/sitemap.xml

An explicit Allow is not required, since anything not disallowed is already permitted. It is worth writing anyway, because it records a decision. The next person to read the file learns that access was considered rather than accidental.

Verify the fix

Changing the code is not the same as fixing the problem. Confirm it.

Fetch robots.txt over HTTPS on the canonical host. A file that only exists on the www or non-www variant is not the file the crawler reads.
Read each user-agent group separately. Rules do not cascade the way people expect, and the most specific matching group is the one that applies.
Compare the status code for a crawler user agent against a browser user agent. A difference means an edge rule, not a robots rule.
Confirm Google-Extended is understood for what it is. It governs Gemini's use of your content and has no effect on Google Search indexing.
Once you have confirmed you are not blocked, stop looking at robots.txt. The next question is whether the crawler can read the page, which is a different lesson.

Check your own file, and check that the crawler agrees

# 1. Read what you actually publish.
curl -s https://example.com/robots.txt

# 2. The rules that matter, isolated.
curl -s https://example.com/robots.txt | \
  grep -iA3 -E "GPTBot|ClaudeBot|PerplexityBot|Google-Extended|User-agent: \*"

# 3. A block can also live at the edge, not in the file.
# A 403 here with a 200 for a normal browser agent is a
# WAF or CDN rule, and robots.txt will never show it.
curl -s -o /dev/null -w "%{http_code}\n" \
  -A "GPTBot/1.1" https://example.com/

curl -s -o /dev/null -w "%{http_code}\n" \
  -A "Mozilla/5.0" https://example.com/

Step 3 is the one people miss. Cloudflare and similar services offer a one-click AI crawler block that never touches robots.txt. If those two status codes differ, the block is at the edge and that is where it has to be changed.

Exceptions and misconceptions

Blocking is a legitimate choice

Publishers, membership sites and anyone whose content is the product have real reasons to disallow AI crawlers. This lesson is not an argument against blocking. It is an argument against believing you blocked something when you did not.

robots.txt is a request, not an enforcement mechanism

Well-behaved crawlers honor it. It is not access control. If content genuinely must not be retrieved, it needs authentication, not a directive.

The block may be at your CDN

Several providers ship a one-click AI crawler block that operates at the edge and leaves robots.txt untouched. A site can be fully permissive in its file and still return 403 to every AI crawler. Test the status codes, do not just read the file.

Being crawled is not being cited

Access is necessary and nowhere near sufficient. Almost every site in our sample allowed the crawlers in, and the same sample had a modal AI readiness score of 2 out of 10. Getting in the door is not the hard part.

Watch it

Primary sources

The measurement above comes from our study, State of AI-Readiness on Business Websites 2026.

Related lessons

Check your own site for this

DeepAudit AI renders your page in a real browser and reports the affected code, so you can see exactly where each finding came from. Free, no signup.

Run a free audit

Last reviewed 2026-09-18. Checks covered: AI Crawler Access.