The short answer.
For AI search in 2026, four technical things matter and one gets far more attention than it deserves. The four are crawler access, content that is readable without running JavaScript, structured data that matches the page, and fast index updates. The overrated one is llms.txt.
If your developer has an afternoon, spend it on the first two. Both are ordinary technical SEO service work, because every AI engine reads the same web pages that search engines do.
What llms.txt is.
llms.txt is a Markdown file that gives AI tools a short summary of a website and a curated list of links to its most useful pages. It was proposed by Jeremy Howard and published on llmstxt.org on 3 September 2024, as “a proposal to standardise on using an /llms.txt file to provide information to help agents use a website.”
The format is simple. The file usually sits at the site root, though the proposal allows one under a sub-path (a file at /docs/llms.txt covers the pages under /docs/). An H1 with the site or project name is the only required part (“the only required section”). After it come, in order: a short summary in a blockquote, optional notes without headings, H2 sections of Markdown links with a note on each, and an “Optional” section for “links an agent can skip when a shorter context is needed.”
It suits one job: helping a coding assistant or agent already working with your documentation find the right page. It was never a ranking signal, and the proposal does not claim to be one.
An annotated example llms.txt.
Here is what a short, honest llms.txt looks like for a B2B services company. It is a fictional firm on example.com; copy the shape, not the words. The notes below explain each part, top to bottom.
example.com/llms.txt (illustrative example)
# Example Co
> Example Co provides managed IT support and cyber security for UK firms of 50-500 staff.
Service pages state scope and response times. Case studies name the sector, not the client.
## Services
- [Managed IT support](https://example.com/managed-it/): scope, hours, response times
- [Cyber Essentials help](https://example.com/cyber-essentials/): what is included
## Evidence
- [Case studies](https://example.com/case-studies/): outcomes by sector
- [About](https://example.com/about/): company facts and locations
## Optional
- [Insights](https://example.com/insights/): guides and articles- H1, the only required line. Use the company name exactly as it appears on your site and in your Organization schema.
- Blockquote summary. One or two plain sentences on who you serve and what you do. Only facts that are visible on your own pages.
- Notes without a heading. Optional context on how your pages are organised.
- H2 link sections. Your most useful pages, each with a short note. Use full, live, indexable URLs.
- “Optional” section. Pages an agent can skip when it is short of space.
Marketing claims are left out on purpose. If the file says something your pages do not, it cannot help and may mislead.
Does llms.txt work? What each AI company says.
As of 13 November 2026, no major AI company documents using other sites’ llms.txt files to choose sources. Google says Search ignores them; OpenAI, Anthropic and Perplexity document robots.txt, not llms.txt, as the control for their crawlers. Measured logs say most files are never requested.
| Who | Has documented | Has not documented |
|---|---|---|
| Google Search | Lists llms.txt among things you can ignore: creating one “will neither harm nor help your site’s visibility or rankings in Google Search, as Google Search ignores them.” Its next-steps list warns against “creating unnecessary AI text files (like llms.txt)”. | Any use of llms.txt in Search, AI Overviews or AI Mode |
| OpenAI | Crawler page names GPTBot, OAI-SearchBot, ChatGPT-User and OAI-AdsBot and says robots.txt controls them. Publishes its own docs index as llms.txt. | Reading other sites’ llms.txt for ChatGPT search or training |
| Anthropic | Help article names ClaudeBot, Claude-User and Claude-SearchBot; robots.txt and Crawl-delay are the controls. | Any mention of llms.txt |
| Perplexity | Crawler page names PerplexityBot and Perplexity-User. Publishes its own docs index as llms.txt. | Reading other sites’ llms.txt to pick sources |
| Chrome Lighthouse | Has an llms.txt audit that flags only server errors; a missing file is “N/A”, as the file is “optional at the moment”. | Any scoring benefit for having one |
What Google staff and server logs add
Google’s John Mueller wrote in April 2025 that “AFAIK none of the AI services have said they’re using LLMs.TXT” and that server logs show “they don’t even check for it”; he compared it to the keywords meta tag. In January 2026, asked whether llms.txt on some Google developer sites was an endorsement, he replied “to be direct, no.” In July 2025 one SEO posted logs showing OpenAI requesting his llms.txt every few minutes; OpenAI did not confirm it.
The largest measurement so far is from Ahrefs, published 15 June 2026. Of 137,210 domains in its web analytics product, about 38,000 had a valid llms.txt, and “97% saw no requests for it whatsoever in May.” Among requests that did arrive, named AI bots made 19.5% and AI search (retrieval) bots just 1.1%. Ahrefs notes that “fetched” does not mean “read” and that its customers skew technical.
So does a B2B site need one?
If you have developer documentation that people use with AI coding tools, yes: it is cheap and can help agents find the right page. If you are a B2B services or software company hoping it will get you into ChatGPT or Google’s AI answers, no amount of effort here will substitute for the work in the chapters below. If someone is selling you an llms.txt file as an AI visibility deliverable, read our guide on what AI search work should cost before you sign.
A copy-ready llms.txt checklist.
If you publish one, paste this list into the ticket.
- ✓File at
https://yourdomain/llms.txt, served as plain text with a 200 status (a server error is the one thing Lighthouse flags). - ✓One H1 with your company name, exactly as on your site.
- ✓A one- or two-sentence blockquote using only facts visible on your pages.
- ✓H2 sections linking your key pages, each with a short note; every URL live, canonical and indexable.
- ✓Nothing listed that robots.txt blocks for the crawlers you want.
- ✓An owner and a review date, so the file is updated when pages move.
- ✓Requests to it counted in your logs after a month, using our AI crawler log file check, so you know whether anything reads it.
Then move on. Everything in the rest of this guide affects whether AI engines can read your pages at all.
Three kinds of AI crawler.
AI companies now run separate bots for three jobs: collecting training data, building a search index, and fetching a page when a user asks. Blocking one does not block the others, and mixing them up is a common technical mistake.
The split matters because the volumes are very uneven. Cloudflare, which sits in front of a large share of websites, reported that in July 2025 training accounted for 79% of AI crawling on its network, search for 17% and user actions for 3.2%.
That is the commercial heart of the robots.txt decision: training crawlers do most of the crawling; search crawlers are the ones that can put your name in an answer. For the traffic side of that trade, see our review of the zero-click and AI Overviews data.
The bots, from each company’s docs.
The table summarises what each company says its bot does and whether robots.txt controls it. Check the official pages before you change anything. For one engine in depth, see our guide to how Perplexity crawls and cites B2B sources.
| Token | Company | What it is for | robots.txt |
|---|---|---|---|
| GPTBot | OpenAI | Training. “Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models.” | Respected |
| OAI-SearchBot | OpenAI | “Used to surface websites in search results in ChatGPT’s search features.” Opted-out sites can still appear as navigational links. | Respected |
| ChatGPT-User | OpenAI | Visits a page when a user asks ChatGPT or a custom GPT. “Not used for crawling the web in an automatic fashion.” | May not apply, as visits are user-initiated |
| OAI-AdsBot | OpenAI | Checks landing pages submitted as ads on ChatGPT | Listed in OpenAI docs |
| ClaudeBot | Anthropic | Collects content that may be used for training; blocking signals future content should be excluded | Respected; Crawl-delay supported |
| Claude-SearchBot | Anthropic | “Navigates the web to improve search result quality for users” | Respected |
| Claude-User | Anthropic | Fetches pages when people ask Claude questions | Respected |
| PerplexityBot | Perplexity | “Designed to surface and link websites in search results on Perplexity”; not used for foundation-model training | Respected |
| Perplexity-User | Perplexity | Visits pages to answer a user’s question | “Generally ignores robots.txt rules” |
| Google-Extended | Controls use of content for training future Gemini models and grounding in some other Google systems. No separate crawler | Respected; does not affect Google Search |
Two lines in Google’s documentation are worth quoting in full, because they settle a common question. Google-Extended “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search.” For AI Overviews and AI Mode, Google’s AI features page points to the normal controls instead: nosnippet, data-nosnippet, max-snippet and noindex. Google also says it intends to support a new training-preference signal (AIPREF) in early 2027. Our guide to optimising for Google AI Overviews covers those controls alongside the eligibility rules.
Writing robots.txt for AI crawlers.
For most B2B companies that want to be named in AI answers, the right setting is to allow the search crawlers and make a deliberate choice about the training crawlers. Do not block everything with “AI” in its name.
Here are two sensible starting points. The first allows everything, which is what you get with no AI-specific rules at all. The second keeps you in AI search while opting out of training.
Option B: in AI search, out of training (example)
# Search and answer crawlers: allowed
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
# Training: opted out
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /Before you use option B, weigh what it costs. Google’s own AI features page says Google-Extended limits “AI training and grounding in some of Google’s other systems”, so blocking it may also limit how Google’s Gemini apps can use your pages. And a company that wants AI models to know what it does in a year’s time has a reason to stay in training data. There is no universally right answer; there is only a choice you make on purpose and write down.
Check the layer above robots.txt
A CDN or firewall can challenge AI crawlers before robots.txt is ever read, which has the same effect as a block. Our guide to AI crawler log file analysis shows how to test for it, and the play on checking AI crawler access gives a short procedure.
JavaScript and rendering.
If the words on your page only appear after JavaScript runs in a browser, many AI crawlers will not see them. Google can render JavaScript; the evidence says most AI crawlers do not.
The most detailed public study is from Vercel, a website hosting company, published 17 December 2024 from traffic on its own network. It found that “none of the major AI crawlers currently render JavaScript”, with Google’s Gemini (which uses Google’s infrastructure) and Applebot as the exceptions. ChatGPT’s and Claude’s crawlers did download JavaScript files, 11.50% and 23.84% of their requests respectively, but did not execute them.
Google’s own JavaScript guidance agrees on the fix. It says “server-side or pre-rendering is still a great idea because it makes your website faster for users and crawlers,” and that “not all bots can run JavaScript.” Vercel’s 2024 data is older, so behaviour may have changed; its advice was to server-render main content, metadata and navigation.
A two-minute test
Open one of your service pages, view the page source (not the inspector), and search for a sentence from the middle of the page. If it is not in the source, a crawler that does not run JavaScript cannot read it. Pricing tables, FAQs and case-study numbers loaded by script are the usual casualties. Our play on server-rendered content walks through it; if a rebuild is needed, see website development.
Schema and structured data.
Structured data helps machines understand a page, but no AI search engine requires a special kind of it. Google says “Structured data isn’t required for generative AI search, and there’s no special schema.org markup you need to add”, and in the same guide calls it “a good idea to continue using it as part of your overall SEO strategy.”
Google’s structured data documentation explains why it still matters: “Google uses structured data that it finds on the web to understand the content of the page.” Google recommends JSON-LD where your setup allows it, and sets one rule that matters more than any other: “don’t add structured data about information that is not visible to the user, even if the information is accurate.”
For Microsoft, Search Engine Roundtable reported in March 2025 that Bing’s Fabrice Canel said at SMX Munich that schema markup helps its large language models understand content. That came via an attendee’s LinkedIn post, not a Microsoft document, so treat it as reported.
What to mark up on a B2B site
- ✓Organization on the home or about page: legal name, logo, URL, and profiles you control.
- ✓Article on insights and guides, with the real publish and update dates.
- ✓BreadcrumbList so the site structure is explicit.
- ✓FAQPage only where the questions and answers are visible on the page.
- ✓Nothing invented. No ratings, prices or awards in markup that are not on the page.
The play on schema that matches the page has a quick test. If an engine already has your facts wrong, see our guide to fixing wrong AI answers about your company.
Bing, IndexNow and Copilot.
Bing deserves more attention than most B2B sites give it, because it now reports when your pages are cited in Microsoft Copilot answers. The two technical levers are accurate sitemaps and IndexNow.
IndexNow is an open protocol that lets a website “instantly inform search engines” when a URL is added, updated or deleted. It is supported by Microsoft Bing, Naver, Seznam.cz, Yandex and Yep, and a submission to one participating engine is shared with the others. Google is not listed among them. You can submit up to 10,000 URLs in one request.
Bing’s webmaster blog, in July 2025, said the lastmod field in your sitemap “remains a key signal” for deciding which URLs to recrawl and reindex, and that the value should reflect the true last change to the page content. It also recommended IndexNow for real-time submission, and said the optional changefreq and priority tags are ignored.
In February 2026 Bing launched AI Performance in Bing Webmaster Tools, in public preview: citations of your pages across Microsoft Copilot, AI summaries in Bing and some partner integrations. It is worth checking monthly; our guide on checking AI search visibility shows where it fits in a baseline.
Priorities for a B2B site.
Do the technical checks in order of what blocks the most. A page that cannot be fetched or read cannot be quoted, however good the writing. For software firms, the writing side is in our guide to SEO for complex B2B and SaaS products.
- 01 · ACCESSSearch crawlers allowed in robots.txt and firewall
- 02 · RENDERKey text present in the page source
- 03 · INDEXPages indexed in Google and Bing
- 04 · MARKUPStructured data matches the visible page
- 05 · FRESHNESSAccurate lastmod; IndexNow on publish
llms.txt, if you want one, comes after all five.
Buyer, in ChatGPT search: “Which UK managed IT providers support 50–200 staff, offer Cyber Essentials Plus help, and publish their response times?”
What decides whether a provider can be named: OAI-SearchBot must be able to fetch the service page; the response times and certification must be in the HTML, not in a script-loaded widget; and the page must be in the index the engine searches. A provider whose firewall challenges bots, or whose service-level table loads by JavaScript, is invisible to that answer however good its service is.
An invented question, written to show the pattern. It is not a real query or answer.
Technical access is necessary, not sufficient. Whether you are then chosen depends on what your pages say and what other sites say about you. Our guides on page patterns AI engines quote and how ChatGPT chooses companies cover those layers. We also ran our own study of where AI answers get their sources.
The technical self-check.
Tick each statement that is true for your website. You get a score out of 100 and the fixes to make first. Nothing you tick is sent anywhere.
Technical score: 0 / 100
Tick what is true today. If you have not checked, leave it unticked.
llms.txt is deliberately not on the list. For help with the checks above, see our technical SEO and content work.
Questions owners ask.
Do I need an llms.txt file?
Not for Google, which says Search ignores the file and it will neither harm nor help. No major AI search engine documents using it to choose sources. It is useful if you have developer documentation used with AI coding tools; otherwise it is low priority.
Does blocking GPTBot remove us from ChatGPT search?
No. OpenAI uses GPTBot for training and OAI-SearchBot for ChatGPT search. Blocking GPTBot signals your content should not be used for training; to stay in ChatGPT search results, allow OAI-SearchBot.
Does blocking Google-Extended affect AI Overviews or rankings?
Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal. It controls use of content for training future Gemini models and grounding in some other Google systems.
Do AI crawlers run JavaScript?
Mostly not, according to Vercel’s December 2024 analysis of its network: none of the major AI crawlers rendered JavaScript, with Google’s Gemini and Applebot the exceptions. Put important text in the HTML that the server sends.
Is there special schema for AI search?
No. Google says there is no special schema.org markup needed for its AI features. Keep normal structured data such as Organization, Article and BreadcrumbList accurate and matching the visible page.
Does Google use IndexNow?
Google is not among the search engines listed as supporting IndexNow. Bing, Naver, Seznam.cz, Yandex and Yep are. For Google, keep sitemaps accurate and use Search Console.
Sources: llmstxt.org; Ahrefs, 15 Jun 2026; Google, AI features guide; Google, AI features; Google, crawlers; Google, JavaScript SEO; Google, structured data; Search Engine Journal, 17 Apr 2025; Search Engine Roundtable, 20 Jan 2026; Search Engine Roundtable, 24 Jul 2025; Search Engine Roundtable, 20 Mar 2025; Lighthouse; OpenAI, crawlers; Anthropic, crawlers; Perplexity, crawlers; Cloudflare, 29 Aug 2025; Vercel, 17 Dec 2024; IndexNow; IndexNow FAQ; Bing, 31 Jul 2025; Bing, 10 Feb 2026. Accessed 10–13 November 2026.