The short answer.
Your access log already records every visit from an AI crawler: who it said it was, which page it asked for and what your server replied. Checking it answers a question no rankings dashboard can: are the AI engines able to read this site at all?
- 01 · COLLECT30 days of access logs, from the server or the CDN
- 02 · FILTERLines whose user agent names an AI crawler
- 03 · COUNTRequests, pages and status codes per bot
- 04 · VERIFYIP addresses against each company’s published list
- 05 · FIXRemove the block, then re-check in a week
The checker in chapter 10 does steps 2 and 3 for a pasted sample.
This is ordinary technical housekeeping, the same kind of work that sits inside our technical SEO services. It comes before content work: a page that cannot be fetched cannot be quoted. We describe what a technical SEO engagement covers in a separate guide.
This guide is about reading what crawlers actually did. Deciding which to allow is covered in our guide to robots.txt rules for AI crawlers and llms.txt.
Why logs beat guesswork.
A log is the only record of what happened between a crawler and your server. Your robots.txt file shows what you asked for. Your analytics shows people. Neither shows whether a bot was let in.
Analytics does not see most crawlers. Website analytics that relies on a tracking script counts a visit only when that script runs in a browser. Vercel, a hosting company, analysed crawler traffic on its network and reported on 17 December 2024 that “none of the major AI crawlers currently render JavaScript”. A bot that never runs your scripts leaves no trace there. It does leave a line in the server log. (People who click through from an AI answer are a separate matter, covered in our guide on following AI referral traffic into pipeline.)
robots.txt is a request, not a gate. A firewall or a bot-protection setting sits in front of your site and can refuse a crawler that robots.txt allows. Cloudflare’s documentation adds that “robots.txt directives are voluntary. Crawlers can ignore them.”
The traffic is large and changing fast. Cloudflare reported on 1 July 2025 that around 30% of global web traffic comes from bots, and that traffic from AI and search crawlers grew 18% between May 2024 and May 2025 across a fixed set of its customers. Within that group, the mix moved towards OpenAI’s bots.
Those are averages across a very large network. They say nothing about your site. Your log does.
Where to find your logs.
Access logs live in one of three places: on the web server, in your hosting control panel, or at the CDN or firewall that sits in front of the site. If you use a CDN, start there: requests it blocks never reach the server’s log.
| Setup | Where to look | Who to ask |
|---|---|---|
| Your own server (Apache or nginx) | The access log file set in the server configuration. nginx writes to logs/access.log in the “combined” format unless told otherwise. | Your developer or system administrator |
| Shared or managed hosting | A “raw access logs” or “logs” section in the hosting control panel, often kept for a limited number of days | Your host’s support team |
| CDN or firewall in front of the site | The provider’s traffic, security events or bot reports. These also show requests that were blocked before reaching your server. | Whoever manages the CDN account |
If you are on Cloudflare, there is a shortcut. Its AI Crawl Control feature, which the documentation says is available on all plans, shows which AI services request your pages. Its metrics include requests over time by crawler, a status code distribution and the most popular paths, and it counts a request as allowed when the response status was 200 to 299.
Ask for 30 days. A single day can mislead, because crawlers do not visit on a schedule. If logs are kept for only a week, download them weekly for a month.
How to read one log line.
Each line in an access log is one request. In the common “combined” format, defined in the Apache documentation, it has the visitor’s IP address, the time, the request, the status code, the reply size, the referring page and the user agent.
Here is an invented line in that format, showing a search crawler being refused:
Illustrative log line (example IP address)
192.0.2.10 - - [10/Nov/2026:06:14:02 +0000] "GET /services/ HTTP/1.1" 403 1042 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot"Two parts do most of the work. The user agent finds the bot; the status code says what happened to it. Here, OpenAI’s search crawler asked for the services page and was refused with a 403.
The bytes figure is a cross-check. A 200 response of a few hundred bytes on a page that should be far larger often means the bot was handed a challenge page instead of your content.
The bot names to search for.
Search for the names the AI companies publish. OpenAI, Anthropic and Perplexity each document their user agents, and each now publishes a list of the IP addresses their bots use. Our guide to how Perplexity finds and cites B2B sources covers its two crawlers in detail.
| Name in the log | Company | Job | Published IP list |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Search: “used to surface websites in search results in ChatGPT’s search features” | openai.com/searchbot.json |
| ChatGPT-User | OpenAI | Fetches a page when a user’s question needs it | openai.com/chatgpt-user.json |
| GPTBot | OpenAI | Training | openai.com/gptbot.json |
| Claude-SearchBot | Anthropic | Search | claude.com/crawling/bots.json (one list for Anthropic’s bots) |
| Claude-User | Anthropic | Fetches a page when someone asks Claude a question | |
| ClaudeBot | Anthropic | Training | |
| PerplexityBot | Perplexity | Search | www.perplexity.ai/perplexitybot.json |
| Perplexity-User | Perplexity | Fetches a page to answer a user’s question | www.perplexity.ai/perplexity-user.json |
| Googlebot | Google Search, which also supplies AI Overviews and AI Mode | Google publishes JSON lists and a reverse DNS method |
Google has no separate AI crawler to look for. Google says a page must be “indexed and eligible to be shown in Google Search with a snippet” to appear as a supporting link in AI Overviews or AI Mode, and that there are “no additional technical requirements”. So Googlebot’s lines are your evidence for Google’s AI features. Google-Extended, the training control, “doesn’t have a separate HTTP request user agent string”, so it will never appear in a log.
OpenAI marks its robots.txt requests. OAI-SearchBot and GPTBot may add a “robots.txt” marker to the user agent when they fetch that file. If you see it, the bot is reading your rules.
The user-triggered bots are the closest thing to a buyer. A ChatGPT-User, Claude-User or Perplexity-User line means a person asked a question and the engine opened your page. Note which pages they request; our guide on how ChatGPT picks companies to recommend explains the searches behind those fetches.
Other names in Cloudflare’s analysis, such as Meta-ExternalAgent, Amazonbot, Bytespider and Applebot, matter more for server load and training policy. Our LLM SEO guide explains how training and live retrieval differ.
Four questions to ask the log.
Ask the log four things about each bot: does it visit, how often, which pages, and what status codes it gets. On a server with a standard log, each answer is one line at the command prompt.
These commands assume the combined format and a file called access.log. No command line? Filter the user agent column in a spreadsheet for the same names, or paste a sample into the checker in chapter 10.
1. Does each bot visit, and how often?
grep -Eio "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User" access.log | sort | uniq -c2. Which status codes does one bot get?
grep -i "OAI-SearchBot" access.log | awk '{print $9}' | sort | uniq -c | sort -rn3. Which pages does it ask for most?
grep -i "OAI-SearchBot" access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -204. Which IP addresses does it come from?
grep -i "OAI-SearchBot" access.log | awk '{print $1}' | sort | uniq -c | sort -rnRun commands 2 to 4 once per bot. In the combined format, field one is the IP address, seven is the page and nine is the status code.
What to write down
Make a small table, one row per bot: requests in 30 days, the share that returned 200, the counts of 403, 429, 404 and 5xx, and the ten most-requested pages. That is your baseline.
Then read the page list with a commercial eye. Are your service and proof pages on it, or is the bot spending its visits on old campaign pages and missing files? Vercel’s analysis found ChatGPT’s crawler spent 34.82% of its fetches on pages that returned 404 and a further 14.36% following redirects, against 8.22% and 1.49% for Googlebot. That is one network in December 2024, but the pattern is worth checking.
If the right pages are fetched and still not quoted, the problem is how the page is written, which our guide to writing passages AI engines can quote covers.
Check the bot is real.
A user agent is only a label, and anyone can send a request labelled GPTBot. Before you act on a line, confirm the request came from an IP address the company has published.
Each company documents how:
- ✓OpenAI publishes a separate JSON file of IP ranges for each bot, and recommends “allowing requests from our published IP ranges” for OAI-SearchBot.
- ✓Anthropic publishes one list and says “If a crawler has a source IP address on this list, it indicates that the crawler is coming from Anthropic.”
- ✓Perplexity publishes a file for each bot and advises site owners to “combine both User-Agent string matching and IP address verification”.
- ✓Google documents a reverse DNS lookup: the IP should resolve to a name ending in googlebot.com, google.com or googleusercontent.com, and back again. It also publishes JSON lists.
The lists are not short and they change. When we fetched them on 12 November 2026, OpenAI’s GPTBot file held 18 IP ranges and was dated 22 September 2026, its OAI-SearchBot file held 39 ranges, and its ChatGPT-User file held 237 and had been updated on 6 November 2026. Perplexity tells site owners to refresh firewall rules from its files automatically.
For a one-off check, take the few IP addresses from command 4 that sent the most requests and look each one up in the company’s file. If they are listed, the traffic is real. If none are, something is borrowing the name; leave its lines out of your counts.
Firewall rules should match both the user agent and the IP range. A rule that allows anything calling itself PerplexityBot is an open door.
What the status codes mean.
The status code is your server’s reply. A 200 means the page was sent; 403 means access was refused; 429 means the visitor was told it was asking too often; 5xx means the server failed. Anything other than 200 on a key page needs an explanation.
| Code | What it means | Likely cause | What to do |
|---|---|---|---|
| 200 | Page sent | Working as intended | Check the byte size looks like a full page, not a challenge page |
| 301 / 302 | Redirected | Old URLs, http to https, trailing slashes | Fine in small numbers. Fix chains |
| 401 / 403 | Refused | Firewall rule, bot protection, country or IP block | Find the rule that fired and add a verified exception for the bots you want |
| 404 / 410 | Not found | Deleted pages, changed URLs, guessed addresses | Redirect those with a replacement |
| 429 | Too many requests | Rate limiting at the server, CDN or a security plugin | Raise or exempt the limit for verified search crawlers |
| 500–504 | Server error | Overload, timeouts, application faults | Fix the fault; compare with crawl bursts |
The standard that defines 429 says the client “has sent too many requests in a given amount of time”; the reply may include a Retry-After header.
Google documents its handling in most detail. It says all 4xx errors except 429 are treated as “the content doesn’t exist”, that pages already indexed which return a 4xx are removed from the index, and that 5xx and 429 responses “prompt Google’s crawlers to temporarily slow down with crawling”. It also warns: “Don’t use 401 and 403 status codes for limiting the crawl rate.”
OpenAI, Anthropic and Perplexity publish no equivalent table, so how each reacts to a run of 403s is not documented. The safe assumption: a page the bot could not fetch is a page it cannot use.
The usual blockers.
When an AI crawler is being refused, the cause is usually a security layer nobody set on purpose, a rate limit, or content that only appears after JavaScript runs.
1. Bot protection at the CDN or firewall
Security products exist to stop automated traffic, and an AI crawler is automated traffic. Cloudflare’s free Bot Fight Mode, for example, “issues computationally expensive challenges” to traffic that matches known bot patterns, and its documentation says the product cannot be adjusted through firewall custom rules. Perplexity’s own setup guide warns that if Bot Fight Mode is enabled, “it can still challenge Perplexity bots” even when an allow rule matches. Anthropic says its bots will not try to get past a CAPTCHA. A challenge page is, in practice, a block.
Defaults also move. On 1 July 2025 Cloudflare announced it would “block AI crawlers accessing content without permission or compensation, by default”. Its documentation, last updated 2 November 2026, now groups AI traffic into Search, Agent and Training and says that for new domains all three “use Allow (do not block) by default”. Open the setting and look.
Cloudflare labels the requests it challenges as “Bot Fight Mode” in its security events, so the cause is easy to confirm.
2. Rate limits
A burst of requests can trip a limit meant for abusive traffic. The sign is a run of 429s after a run of 200s from the same IP address. Exempt verified search crawlers, or raise the limit. Anthropic also supports the Crawl-delay line in robots.txt.
3. Content that needs JavaScript
This one is not an error. The log says 200, but the HTML sent was a shell and a script, and the words arrive later in a browser. Crawlers that do not run JavaScript receive the shell. View the page source and check your key sentences are in it. The play on server-rendered content gives the test in full.
4. Rules written long ago
An old robots.txt line, a country block or a strict security plugin can refuse crawlers you now want. After a robots.txt change, OpenAI says its search systems take about 24 hours to adjust, and Perplexity says up to 24 hours.
5. Leftovers from a redesign
Crawlers keep asking for addresses they learned earlier. After a migration, a high share of 404s usually means old URLs were never redirected. Our website redesign checklist covers the redirect plan.
Buyer, in ChatGPT search: “Which UK calibration labs are UKAS accredited for pressure gauges and publish their turnaround times?”
What the log would show for a lab that is left out: OAI-SearchBot requesting the accreditation page and the services page, and receiving 403 each time from a firewall rule added two years ago. The pages are good; the engine was never allowed to read them. Analytics would not show it. Three log lines would.
An invented question and scenario. Not a real query, answer or company.
The log-line checker.
Paste lines from your access log below. The checker finds the crawler names, counts the status codes each received and points out likely blocks. It runs in your browser only; nothing you paste is sent anywhere.
Illustrative example: the sample lines are invented.
No lines checked yet.
The checker reads the name in each line. Names can be faked, so confirm the IP addresses against the published lists (chapter 7) before you change any rule.
A pasted sample is a quick read, not a verdict; run the counts in chapter 6 across a full month.
After the check.
Fix one blocker at a time, wait a week, and read the log again. The proof that a fix worked is the same bot, on the same pages, now receiving 200.
A simple routine
- ✓Week 1: build the baseline table and verify the busiest IP addresses.
- ✓Week 2: remove the largest block. Note the date and the rule you changed.
- ✓Week 3: re-run the counts. Redirect the old URLs that bots keep requesting.
- ✓Monthly: repeat the four questions, and again after any hosting, CDN or plugin change.
What the log cannot tell you
A clean log proves access, not that you are being named. Once the bots are getting 200s, test the outcome directly: our guide to checking your AI search visibility by hand gives a method, and the free AI visibility report runs a sample of buyer questions for you.
The log also covers only your own site. AI answers draw on many other sources, as our study of where AI answers get their sources shows.
Where this sits in the wider work
Access is the first layer. Pages that answer buyer questions, proof that other sites repeat, and measurement tied to enquiries come next. That is the scope of our GEO and AEO programme. The crawl-access and indexing fixes this guide turns up belong in ongoing SEO for B2B websites.
Questions owners ask.
How do I check if ChatGPT crawls my website?
Search your access log for three names: OAI-SearchBot (ChatGPT search), ChatGPT-User (a page fetched for a user’s question) and GPTBot (training). Then check their status codes: 200 means the page was sent; 403 or 429 means the bot was turned away.
Are AI crawlers visiting my site if I see nothing in GA4?
They may be. Script-based analytics generally does not record crawlers, since Vercel’s December 2024 analysis found the major AI crawlers do not run JavaScript. Look in server or CDN logs.
How do I know a request really came from GPTBot or ClaudeBot?
Compare the IP address in the log line with the list the company publishes. OpenAI and Perplexity publish a file for each bot; Anthropic publishes one list. A matching user agent alone is not proof.
Why is OAI-SearchBot getting 403 when robots.txt allows it?
Because robots.txt is not what refused it. A 403 comes from a firewall, bot protection, a security plugin or a hosting rule. Find the rule in the security event log and add an exception that matches both the user agent and the published IP ranges.
How much AI bot traffic is normal?
There is no standard figure for a single site. Cloudflare’s network-wide data shows the mix changing quickly. Track your own month-on-month trend instead of comparing against an average.
Sources: OpenAI, crawlers; GPTBot IPs; OAI-SearchBot IPs; ChatGPT-User IPs; Anthropic, crawlers; Anthropic IPs; Perplexity, crawlers; Google, verifying crawlers; Google, HTTP status codes; Google, AI features; Google, common crawlers; Cloudflare, 1 Jul 2025; Cloudflare press release; Cloudflare, AI crawler policies; Cloudflare, Bot Fight Mode; Cloudflare, AI Crawl Control; AI Crawl Control metrics; Apache, log files; nginx, log module; RFC 6585; Vercel, 17 Dec 2024. All accessed 12 November 2026.