Why this comes before everything
If a crawler cannot fetch a page, nothing else you do will register. Content quality, schema, internal linking — all irrelevant to a page that is never retrieved. This is also the step most often skipped, because the failure is invisible: nothing on the site looks broken.
The whole check takes about ten minutes. Run it before committing budget, not after.
Step 1: Confirm the main AI crawlers are allowed
Open robots.txt and look for the major AI crawler user-agents. The names have changed over time and differ by vendor, so check the current list rather than an old article's. What you are looking for is whether any of them sit inside a Disallow block.
- Check the AI user-agents specifically, not only the wildcard rule
- A Disallow: / under User-agent: * blocks every AI crawler, including ones you want
- Watch for path patterns written for search crawlers that also catch the AI ones
Step 2: Check what a crawler actually receives
Allowed in robots.txt is necessary but not sufficient. The next failure is that the request returns something other than your content — a challenge page, a consent interstitial, a geo-block, or an empty shell that fills in only when JavaScript runs.
Fetch one of your most important pages with JavaScript disabled, or inspect the raw HTML response. If the body contains the real text, you are fine. If it contains a script tag and little else, that is the problem.
- Cloudflare and similar bot challenges are the most common accidental block
- Consent walls that render before content hide the page from retrieval
- Some retrieval systems do not execute JavaScript at all
Step 3: Confirm your key pages are reachable from the site itself
The pages that matter most are the ones that answer your buyers' questions: pricing, product scope, FAQ, comparison pages. If they are orphaned, blocked by a noindex directive, or reachable only through a JavaScript menu, they will not be found.
Sitemaps help here, but a sitemap listing a page does not make it indexable — check the page itself for a noindex tag.
Step 4: Look for contradictions about your own facts
Retrieval systems assemble answers from whatever they find. If your pricing page and a three-year-old press release disagree, the assembled answer will pick one, and you do not control which. This step is quick and often more damaging than anything technical.
- Compare prices, product names and leadership across the site
- Check that old landing pages are not still live with stale claims
- Where facts cannot be corrected, add a dated note saying what is current
What to do with a failure you find
Fix crawl access immediately and before anything else. It is cheap, it is verifiable, and every other investment depends on it. Record the before-and-after: which user-agent was blocked, what the raw response looked like, what it looks like after.
That record matters later, because when a visibility result takes six weeks to appear, the first question is whether the fundamentals were ever right.
Run this before the first content brief, not after the first disappointing report. Ten minutes here can save months of work that was never going to land. Sources reviewed on 2026-10-17. This article records a method with verifiable boundaries; it does not claim that any search or AI platform will index, cite or recommend anything.