> ## Content Index > Fetch the complete content index at: https://theseguysknow.io/llms.txt > Use this file to discover other available public pages before exploring further. # Which AI Crawlers Should Publishers Allow or Block in 2026? - URL: https://theseguysknow.io/ai-crawlers-robots-txt/ - Published: 2026-09-20T11:16:57.000Z - Updated: 2026-09-20T11:16:57.000Z - Description: AI companies now use separate crawlers for search, model training and pages requested by users. We checked the current rules from OpenAI, Anthropic, Perplexity and Google to see what publishers can block without giving up useful AI visibility. - Author: Larson Bryce - Tags: AI & Tech, AI Tools & Models If you copy a generic “block AI bots” robots.txt file in 2026, there is a good chance it blocks more than you intended. OpenAI and Anthropic now use different agents for search and model training, Perplexity says its search bot is not a training crawler, and Google handles AI Overviews through a separate Search Console control rather than `Google-Extended`. We checked the current documentation from all four companies to map what each control does, whether it affects AI search visibility, and what you give up by blocking it. ## Quick Answer Publishers who want their articles discoverable in AI search while limiting separately controllable model-training crawlers can currently make that distinction with OpenAI and Anthropic. OpenAI uses `OAI-SearchBot` for ChatGPT Search and `GPTBot` for content that may be used to train its foundation models, while Anthropic separates `Claude-SearchBot`, `ClaudeBot` and `Claude-User`. Perplexity says `PerplexityBot` is designed to surface and link websites in its search results and is not used to collect content for foundation-model training. Google is less straightforward because `Google-Extended` controls certain Gemini training and grounding uses together, while AI Overviews, AI Mode and generative features in Discover now have their own Search Console setting. | Company | Control | Main purpose | What happens if you block it | | ---------- | ---------------------------- | ------------------------------------------------------ | -------------------------------------------------------------------------------------- | | OpenAI | OAI-SearchBot | ChatGPT Search discovery | Pages stop appearing in ChatGPT search answers, apart from possible navigational links | | OpenAI | GPTBot | Content that may be used for foundation-model training | Signals that site content should not be used for that training | | OpenAI | ChatGPT-User | Pages fetched because a user requested them | Not the control for ChatGPT Search; robots.txt may not apply | | Anthropic | Claude-SearchBot | Claude search | May reduce visibility and accuracy in Claude search results | | Anthropic | ClaudeBot | Content that may contribute to model training | Future material should be excluded from Anthropic training datasets | | Anthropic | Claude-User | User-requested page retrieval | Claude cannot retrieve the site for those user requests | | Perplexity | PerplexityBot | Perplexity search discovery | Works against visibility in Perplexity search | | Perplexity | Perplexity-User | User-requested page retrieval | Generally not controlled by robots.txt | | Google | Google-Extended | Certain Gemini training and grounding uses | Restricts both covered uses without affecting normal Google Search | | Google | Search generative AI control | AI Overviews, AI Mode and generative Discover | Removes the site from those features while leaving conventional Search available | **Checked against first-party documentation on 20 September 2026.** ## Search visibility and model training are already separate choices OpenAI provides the cleanest example of why treating every AI crawler as the same thing no longer works. Its documentation says `OAI-SearchBot` is used to surface websites in ChatGPT Search, while `GPTBot` crawls content that may be used in training OpenAI’s generative foundation models. The controls are independent, and OpenAI explicitly gives publishers the option of allowing its search bot while blocking GPTBot. A publisher that wants ChatGPT Search visibility without allowing GPTBot can therefore use: ```text User-agent: OAI-SearchBot Allow: / User-agent: GPTBot Disallow: / ``` That is a useful distinction for a publication because search discovery can produce a source link and referral traffic, while allowing material to be collected for future model training is a separate decision. Blocking both because they belong to OpenAI removes that choice and can cost the site visibility it may have wanted to keep. Anthropic now offers a similar split. `Claude-SearchBot` is used to improve Claude’s search results, `ClaudeBot` collects public web content that could potentially contribute to model training, and `Claude-User` handles pages retrieved because somebody asked Claude to access them. Anthropic says all three respect robots.txt and explains the consequence of disabling each one separately. For a publisher interested in Claude search visibility but unwilling to allow its training crawler, the configuration can therefore be selective: ```text User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / User-agent: ClaudeBot Disallow: / ``` This is where a long list titled “AI bots to block” starts becoming less useful than it looks. `GPTBot` and `OAI-SearchBot` belong to the same company but represent different choices, just as `ClaudeBot` and `Claude-SearchBot` do. ## User-requested page visits create a third category Search crawling and training are not the whole picture because AI assistants can also visit a webpage after somebody explicitly asks them to. OpenAI uses `ChatGPT-User` for certain actions initiated through ChatGPT and Custom GPTs. It is not used to decide whether a page can appear in ChatGPT Search, and OpenAI says robots.txt rules may not apply because the request originates from a user rather than an automatic crawl. Anthropic takes a different approach. Its `Claude-User` performs a comparable job, but Anthropic says its bots honor robots.txt, allowing a publisher to prevent Claude from retrieving the site in response to user-directed requests. Blocking it may therefore reduce visibility in cases where Claude needs to open the page for somebody. Perplexity separates these jobs as well. `PerplexityBot` is its search crawler, which Perplexity says is designed to surface and link websites in search results and is not used to collect content for foundation-model training. `Perplexity-User` can visit a page while answering a user’s question, may include a link to that page in the response and is also not used for foundation-model training. The important difference is that Perplexity says `Perplexity-User` generally ignores robots.txt because the fetch was requested by a user. That puts it closer to OpenAI’s treatment of `ChatGPT-User` than Anthropic’s treatment of `Claude-User`. This comparison is useful when writing an access policy because a user-requested fetch is not simply another version of a search crawler. The same type of request is treated differently by three major AI companies, so a robots.txt rule that works for Claude does not necessarily produce the same result in ChatGPT or Perplexity. ## Google does not fit the same crawler model Google makes the comparison less tidy because `Google-Extended` looks like another crawler when written inside robots.txt: ```text User-agent: Google-Extended Disallow: / ``` Google describes it as a standalone **product token**, however, and says there is no separate Google-Extended HTTP user-agent string. Crawling happens through existing Google user agents, while the token controls whether content Google has crawled may be used for certain Gemini purposes. Those purposes currently include training future generations of Gemini models that power Gemini Apps and the Vertex AI API for Gemini, along with grounding in Gemini Apps and Grounding with Google Search on Vertex AI. Google states that `Google-Extended` does not affect normal Google Search inclusion or rankings. The awkward part for publishers is that training and those grounding uses share the same control. OpenAI lets you block `GPTBot` while leaving `OAI-SearchBot` available, and Anthropic lets you separate `ClaudeBot` from `Claude-SearchBot`; Google does not currently document an equivalent way to disable only the training part of `Google-Extended` while retaining every grounding use covered by the token. Google’s own generative Search products are now handled separately. Since 31 August 2026, Search Console has allowed verified site owners to exclude a property from AI Overviews, AI Mode and generative AI features in Discover while keeping the site eligible for conventional Google Search. Google also states that this setting does not control AI training and directs publishers to `Google-Extended` for that decision. We have a separate guide to [opting out of Google AI Overviews without blocking normal Google Search](https://theseguysknow.io/opt-out-google-ai-overviews/?utm%5Fsource=chatgpt.com), including what the setting removes, what remains available to Google and how the new generative AI performance reports can help publishers see what they would be giving up. For this article, the relevant difference is that Google now gives a publisher several independent decisions about the same public page: it can remain in ordinary Search, be excluded from Google’s generative Search features, and have separate restrictions applied through `Google-Extended`. ## What we would allow on a publisher site For a publication that wants its public articles discoverable in ChatGPT, Claude and Perplexity while declining the training crawlers that can be separated cleanly, a reasonable starting configuration looks like this: ```text # OpenAI search User-agent: OAI-SearchBot Allow: / # OpenAI model-training crawler User-agent: GPTBot Disallow: / # Anthropic search User-agent: Claude-SearchBot Allow: / # Anthropic user-requested retrieval User-agent: Claude-User Allow: / # Anthropic model-training crawler User-agent: ClaudeBot Disallow: / # Perplexity search User-agent: PerplexityBot Allow: / ``` There is no equivalent Perplexity training rule in its current crawler documentation because the company says `PerplexityBot` is not used for foundation-model training. Blocking it for that reason would therefore remove a search-discovery route without blocking the use you were trying to prevent. I would also avoid inserting `Google-Extended: Disallow` into a generic publisher template without explaining its wider effect. A site owner may be happy to block the covered Gemini training use but still care about Gemini grounding, and Google currently places both under the same token. Whether that trade-off makes sense depends on what the publisher wants from Gemini rather than on a blanket rule about AI companies. The same caution applies to user-requested fetchers. `Claude-User` can be managed through robots.txt according to Anthropic, `ChatGPT-User` may not follow those rules according to OpenAI, and `Perplexity-User` generally ignores them according to Perplexity. There is no honest universal configuration that treats all three as interchangeable. ## robots.txt may not be the thing blocking the crawler Even a correct robots.txt file does not guarantee an AI service can reach the site. A CDN, firewall or bot-protection rule can reject the request before the crawler receives useful content, which is why OpenAI and Perplexity publish current IP ranges for their agents and Perplexity recommends combining user-agent matching with IP verification when configuring a WAF. I would not copy those IP ranges into a static article because the providers update them. Their live lists are the better source when somebody is configuring Cloudflare, AWS WAF or another security layer. It is also worth separating crawler policy from access control. robots.txt tells compliant automated systems how a site wants them to crawl; it does not make a public URL private. If a page contains something that strangers should not be able to retrieve, authentication and real access controls matter more than which AI user-agent has been disallowed. That distinction is similar to the terminology problem we covered in [AI Model vs Tool vs Platform vs Provider](https://theseguysknow.io/ai-model-vs-tool-vs-platform-vs-provider/?utm%5Fsource=chatgpt.com). The visible AI product is only one layer of the system, and the bot touching your website may have a much narrower job than the company name suggests. ## Final Verdict A publisher deciding what to do with AI crawlers in 2026 should write the policy around **uses rather than companies**. OpenAI and Anthropic currently provide separate controls for search discovery and model-training crawlers, which makes it possible to keep the former while blocking the latter. Perplexity says its documented search and user-requested agents are not used to collect content for foundation-model training, while Google separates its generative Search controls from `Google-Extended` but still groups certain Gemini training and grounding uses under the same token. For a publication that values citations and referral traffic, blocking every AI user-agent by default is therefore a much broader decision than it first appears. The more precise approach is to decide separately whether you want search discovery, user-requested retrieval and model-training collection, then use the controls each company currently provides for those specific jobs. The crawler table and policies in this article were checked against first-party documentation on **20 September 2026** and should be treated as a maintained reference because the companies can change their agents and rules. #### Sources [OpenAI — Overview of OpenAI Crawlers](https://platform.openai.com/docs/bots?ref=theseguysknow.io) [Anthropic — Web crawling and crawler controls](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler?utm%5Fsource=chatgpt.com) [Perplexity — Perplexity Crawlers](https://docs.perplexity.ai/guides/bots?ref=theseguysknow.io) [Google — Google-Extended and common crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers?ref=theseguysknow.io) [Google Search Console — Search generative AI control](https://support.google.com/webmasters/answer/16908024?hl=en&utm%5Fsource=chatgpt.com)