Managing AI crawlers:block training,stay visible

AI crawlers are managed in robots.txt one by one, according to their purpose: you can block training crawlers such as GPTBot or ClaudeBot without shutting out the search crawlers, OAI-SearchBot, Claude-SearchBot and PerplexityBot, that decide whether you show up in AI search. The major providers now separate crawlers that collect content for model training, crawlers that build a search index and fetches triggered by a user in a chat. Block everything that sounds like AI and you disappear from the answers your customers read when they ask for a supplier. Leave everything open and your content is available for training without anyone asking you. The sensible path lies in between, and it takes only a few lines to put in place.

Nikolai Schöbel and Jeremias Burger

Co-founders of Scalableloops GmbH. Nikolai Schöbel leads online marketing and AI strategy, Jeremias Burger the AI architecture. Both build AI systems and train teams on them in their own agency work.

Which AI crawlers exist and what are they used for?

The overview below summarises what OpenAI, Anthropic, Perplexity and Google say about their crawlers in their own documentation (as of 25 September 2026). Providers update these pages from time to time, so check the original before you change your robots.txt.

OpenAI states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, while disallowing GPTBot indicates that content should not be used to train its foundation models. Anthropic draws the same line: blocking ClaudeBot excludes future content from training, and blocking Claude-SearchBot prevents indexing for search, which Anthropic says may reduce visibility in search results. Perplexity says it does not run a training crawler; PerplexityBot exists to surface and link websites in Perplexity's search results.

Should you block AI crawlers or let them in?

There is no single answer, but the question splits neatly into two separate decisions. The first is about training: may your content feed into future models? The second is about visibility: do you want to appear as a source in AI search answers? Because providers run separate crawlers for each, you can answer the two independently.

For businesses that want to be found through AI answers, search is the more important of the two. The overview shows which stance fits which goal.

1. 01

   01

   Visible, training allowed

   Allow every crawler. This fits when your content is meant to promote you publicly anyway and you see no reason to keep it out of training.

2. 02

   02

   Visible, training excluded

   Block GPTBot, ClaudeBot and Google-Extended, allow OAI-SearchBot, Claude-SearchBot and PerplexityBot. This fits when you want to protect your texts but still be named in AI search.

3. 03

   03

   Protect specific areas

   Block individual directories for all AI crawlers, such as customer areas, dealer price lists or internal documents, and leave the rest open. For truly confidential content robots.txt is not enough, as explained below.

4. 04

   04

   Opt out completely

   Block every AI crawler. This is a deliberate choice with a cost: your business is missing as a source in AI search answers when customers ask there for a supplier.

Do small and mid-sized business websites block AI crawlers at all?

None of the websites we examined block the crawlers that ChatGPT, Claude or Perplexity use to read web pages for their search. Crawlers are programs that fetch web pages automatically. On 26 Sep 2026 we checked the robots.txt of 22 websites of small and mid-sized German businesses. The robots.txt is a small file on a website that tells crawlers which pages they may fetch. We evaluated it according to the RFC 9309 standard. Put simply: if the file names a crawler, its instructions apply to that crawler, otherwise the general ones do. And the most specific rule takes precedence.

17 of the 22 websites have a robots.txt. Only two of them name an AI crawler at all, meaning a program that AI services use to read web pages. Only one website blocks one of these crawlers: CCBot, the crawler of the Common Crawl archive. GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot and Google-Extended are allowed on all 22 websites. In most cases this is simply because the robots.txt does not mention them.

So if an AI service does not fetch your pages, this survey suggests the robots.txt is probably not the reason. That is why it pays to look at your firewall and bot protection. More on that in the next section.

One more finding: six of the 22 websites offer an llms.txt. This is a file that gives AI systems a short overview of the most important content on a website. One of these files is generated automatically by an SEO plugin for WordPress. What this file does and does not do is explained in our article on llms.txt. The sample is not representative. It comes from our client base and from businesses whose website we reviewed for a proposal. We look after some of these websites ourselves.

Where does robots.txt reach its limits with AI crawlers?

robots.txt is not access control. The RFC 9309 standard states plainly that its rules are not a form of access authorization, and Google notes that the file cannot enforce crawler behaviour. Anything genuinely confidential belongs behind a password, not behind a line in robots.txt.

Then there are fetches triggered by users. OpenAI's documentation says that robots.txt rules may not apply to ChatGPT-User because a person initiated the action; according to OpenAI, ChatGPT-User does not determine whether content can appear in ChatGPT search. Perplexity says Perplexity-User generally ignores robots.txt. Anthropic, by contrast, says it respects robots.txt and offers a separate block for Claude-User.

Declared crawlers are not free of dispute either. In August 2025 Cloudflare accused Perplexity of switching to undeclared crawlers with rotating addresses when blocked; Perplexity rejected the accusations. The takeaway: robots.txt is the right tool to state your wishes to reputable providers. If you must make sure content is not fetched at all, you need technical barriers. For a view of your settings in the wider context of AI visibility, talk to our GEO agency.