JavaScript and AI Crawlers

What we measured

25.0% of sites hid at least one answer-critical element from a non-rendering AI crawler. Body text was the most commonly hidden element at 15.5%, followed by name, address and phone at 14.7%, internal links at 13.9%, and the contact or booking form at 11.1%. The predictor was not whether a site was custom built. It was whether the platform server renders answer-critical content by default: 4.0% of managed server-rendered builds were affected against 28.3% of everything else, a two-proportion z of 3.69 at p below .001.

Why it matters

• The measure is first-party crawl visibility to non-rendering crawlers such as GPTBot, ClaudeBot and PerplexityBot. It is not a claim that an assistant cannot know you exist. Assistants also read search indexes and retrieval layers, and those may have rendered your page even when a direct fetch did not.
• What it does mean is that the copy you control, on the domain you own, is not what those crawlers received. Every other source describing you gets to speak instead.
• It fails silently. Nothing errors, nothing 404s, and a browser test looks perfect, because a browser is the one client that does run the JavaScript.
• The elements that go missing are the expensive ones. Body text, contact details and the booking form are precisely what somebody is trying to find when they ask an assistant about a business.

The failure

"use client";

export default function Services() {
  const [services, setServices] = useState([]);

  useEffect(() => {
    fetch("/api/services")
      .then((r) => r.json())
      .then(setServices);
  }, []);

  return <ul>{services.map((s) => <li key={s.id}>{s.name}</li>)}</ul>;
}

The fix

// A server component. The fetch happens before the HTML is sent,
// so the list is in the markup every client receives.
export default async function Services() {
  const services = await getServices();

  return <ul>{services.map((s) => <li key={s.id}>{s.name}</li>)}</ul>;
}

Verify the fix

Changing the code is not the same as fixing the problem. Confirm it.

1. Disable JavaScript in your browser and reload the page. This is the thirty second version of the test and it catches most of it.
2. Fetch the page with curl and a crawler user agent, then search the output for text you know is visible in a browser. A match means the content is in the HTML. No match means it is not.
3. Test the pages that answer questions, not the homepage. Services, pricing, contact and location pages are where the answer-critical content lives.
4. After changing to server rendering, run the same curl and confirm the string is now present. The fix is only real when the raw fetch shows it.
5. Check your name, address and phone specifically. It was invisible on 14.7% of the sites we tested and it is the single element most likely to be asked for.

# 1. What a non-rendering AI crawler receives.
curl -s -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); \
compatible; GPTBot/1.1; +https://openai.com/gptbot" \
  https://example.com/services > raw.html

# 2. Is the answer in there? Search for text you know is on the page.
grep -c "Emergency callout" raw.html

# 3. Compare against the rendered DOM.
npx -y puppeteer-cli screenshot --help >/dev/null 2>&1
node -e '
const puppeteer = require("puppeteer");
(async () => {
  const b = await puppeteer.launch();
  const p = await b.newPage();
  await p.goto("https://example.com/services", {waitUntil: "networkidle0"});
  const html = await p.content();
  require("fs").writeFileSync("rendered.html", html);
  await b.close();
})();'

# 4. The gap between these two numbers is your exposure.
grep -c "Emergency callout" rendered.html

Exceptions and misconceptions

Watch it

Primary sources

• OpenAI: GPTBot and how it crawls
• Google Search Central: understand JavaScript SEO basics
• MDN: server-side versus client-side rendering

The measurement above comes from our study, AI Visibility Study 2026.

Related lessons

• Blocking AI Crawlers
• AI Readiness

Last reviewed 2026-09-18. Checks covered: .