ProlinkageGet your free report
Home/Resources/AI Shortlist Plays
AI Shortlist Plays / Access and structure

Can AI crawlers actually reach our website?

Open your robots.txt, check your CDN and firewall settings, and test what each AI crawler gets back. If the search crawlers cannot fetch your pages, nothing else in these plays can work.

By ProlinkagePublished 8 November 2026Play 26 of 40 · Edition 1

Effort: 1 to 2 hours, plus an hour of developer time

Who does it: Marketing lead, with your web developer

Best for: Any B2B firm, especially behind Cloudflare or another CDN

Evidence: Strong (OpenAI crawler documentation, read 8 November 2026)

01 / WHY IT MATTERS

Why it matters.

OpenAI states that “sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers”, and that a robots.txt change takes about 24 hours to take effect (OpenAI crawler documentation, read 8 November 2026).

Blocks also happen outside robots.txt. On 1 July 2025 Cloudflare changed its default to block AI crawlers unless site owners allow them. A firewall or bot-protection setting can send crawlers an error while every human visitor sees the page normally.

02 / SELF-CHECK

The 10-minute self-check.

Three checks, ten minutes:

  • Open yourdomain.com/robots.txt and search for OAI-SearchBot, GPTBot, PerplexityBot, ClaudeBot, Google-Extended and any blanket “Disallow: /”.
  • Ask your developer to fetch your homepage with each crawler’s user agent and report the status code.
  • Open your CDN or firewall dashboard and find the setting for AI bots.

Anything other than a 200 for the search crawlers needs a decision this week.

03 / STEPS

Steps.

  1. Decide your policy. Allow the search crawlers (OAI-SearchBot, PerplexityBot). Training crawlers (GPTBot, Google-Extended) are a separate choice: OpenAI says blocking GPTBot does not remove you from search answers.
  2. Write the policy as explicit robots.txt rules, one block per crawler.
  3. Make the CDN and firewall settings match the policy.
  4. Re-test the status codes. A user-agent test is a good first check, though real crawlers also come from published IP ranges.
  5. Re-check after any hosting, CDN or security plugin change, and log the date.

It is the first check in our technical SEO reviews.

04 / OUR OWN SITE

What we saw on prolinkage.com.

05 / MEASURE

How to measure it.

  • Status codes: one line per crawler, monthly and after every infrastructure change.
  • Crawler visits: hits from OAI-SearchBot and PerplexityBot in your server logs, if you can read them.
  • Outcome: sessions in GA4’s AI Assistant channel (play 36).
07 / SOURCES

Sources.

AI Shortlist Plays

The self-check shows what you can see yourself.

What the engines say across repeated runs, which sources they read and who they name instead is harder to see from inside. Our free AI Visibility Snapshot runs a sample of your buyers’ questions and is reviewed by a person.

More than a decade of experience, across businesses and markets.