AI crawlers: the complete list, and how to allow or block them

An AI crawler is a bot that an AI company sends to read web pages, either to answer a user's question in real time or to collect training data for its models. You control which ones may read your site with a few lines in robots.txt. The catch is that every AI company runs more than one crawler, and blocking the wrong one can take your site out of ChatGPT, Claude or Perplexity answers while doing nothing to stop training.

To see what your own site allows right now, run it through the free AI crawler checker. It reads your robots.txt the way the crawlers do and shows the exact line behind every verdict.

What are the two kinds of AI crawlers?

AI crawlers fall into two groups that do very different jobs. Search crawlers fetch your pages so an assistant can quote and link to them in its answers, while training crawlers collect content to build future AI models. Blocking a search crawler removes you from that assistant's answers; blocking a training crawler only opts you out of training.

There is also a third group worth knowing about: user-triggered fetchers such as ChatGPT-User and Perplexity-User. They open a page only when a person asks the assistant about it or pastes a link.

The complete list of AI crawlers

These are the user-agent names to use in robots.txt. Names are not case-sensitive.

Google-Extended and Applebot-Extended are not separate crawlers. They are control names: Googlebot and Applebot still visit your pages, and these names only decide whether the content may be used for AI training.

Does blocking GPTBot remove my site from ChatGPT?

No. GPTBot only collects content for training OpenAI's models. ChatGPT finds and cites pages through OAI-SearchBot, so a site that blocks GPTBot but allows OAI-SearchBot can still be shown and linked in ChatGPT answers. Block OAI-SearchBot and ChatGPT can no longer cite you, whatever you do with GPTBot.

Does blocking Google-Extended remove me from AI Overviews?

No. Google-Extended controls whether Google may use your content for Gemini training and grounding, not whether you appear in Google Search. AI Overviews and AI Mode are built from the normal Googlebot index, so the only way to keep a page out of them is to keep it out of Google Search, with a noindex or nosnippet rule.

Should you block AI crawlers?

It depends on whether people need to find you or whether your content is the thing you sell. For most businesses, the best setup is to allow the AI search crawlers so assistants can recommend and cite you, and decide separately about training.

Whatever you choose, make it a decision. In our research, many blocks looked like accidents: a block list copied from somewhere, a security plugin's default, or a CDN setting nobody remembered turning on.

How do you block or allow AI crawlers in robots.txt?

Add a group to the robots.txt file at the root of your site. Each group names one or more crawlers on User-agent lines, followed by Allow or Disallow rules. Here are the three setups most sites need. Keep your existing rules and make sure each crawler is named in only one group.

Visible in AI answers, no AI training (the usual choice for businesses):

# AI assistants may read and cite the site
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /

# No AI model training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
User-agent: Bytespider
Disallow: /

Block every AI crawler:

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: GPTBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: ClaudeBot
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
User-agent: Bytespider
Disallow: /

This does not affect Google Search or AI Overviews, which use Googlebot.

Block AI crawlers from one part of the site only:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /premium/
Disallow: /members/

The AI crawler checker also generates these groups for you, ready to copy.

How do you check if your site blocks AI crawlers?

The quickest way is to enter your site in the free AI crawler checker, which tests 15 AI crawlers and shows which assistants can cite you. To check by hand, open yoursite.com/robots.txt and look for each crawler's name.

When you read the file yourself, remember three rules that catch people out:

1. A crawler follows only the most specific group that names it. If GPTBot has its own group, the rules under User-agent: * no longer apply to it.
2. A crawler with no group of its own follows User-agent: *. So User-agent: * with Disallow: / blocks every AI crawler you have not named.
3. The longest matching rule wins. Allow: /blog/ beats Disallow: / for pages under /blog/.

Why robots.txt is not always the whole story

A firewall or CDN can turn AI crawlers away before they ever read robots.txt. Several CDNs offer a one-click "block AI bots" setting, and security plugins sometimes block unfamiliar bots by default. In that case your robots.txt can say Allow while the crawler is refused at the door.

The AI crawler checker includes a firewall test for this: it requests your page as a normal browser and then under each AI crawler's name, and flags any crawler that is refused. If it flags one, check your CDN's bot settings.

The reverse is also true. robots.txt is a request, not a lock. The major AI companies say their crawlers respect it, though some say their user-triggered fetchers may not, because a person asked for the page. If content must stay private, put it behind a login.

What we found on Indian websites

In October 2026 we ran the AI crawler check on 80 well-known Indian websites, 20 each from news, edtech, universities and government. Sites whose robots.txt could not be read are left out of each count.

The pattern is clear. News publishers are protecting their content, and most of those that block GPTBot also block OAI-SearchBot and ChatGPT-User, which takes them out of ChatGPT's cited answers, not just out of training. Education websites are almost entirely open. Most government websites have never set up a working robots.txt at all.

Common mistakes with AI crawlers

• Blocking GPTBot and thinking you have left ChatGPT. You have only opted out of training. OAI-SearchBot decides whether ChatGPT cites you.
• Blocking OAI-SearchBot and PerplexityBot by accident. Copied block lists often mix search and training crawlers. You lose AI citations and gain nothing.
• Expecting Google-Extended to remove you from AI Overviews. It does not. AI Overviews come from Googlebot.
• Naming a crawler in two groups. Only one group applies, which makes the result hard to predict. Keep each name in one place.
• A robots.txt that returns an error. When the file cannot be read because of a server error, crawlers treat the whole site as blocked until it can be read again.
• Forgetting the CDN. A firewall rule can block AI crawlers whatever robots.txt says.

How long does a robots.txt change take to work?

Usually about a day. Crawlers cache robots.txt and re-read it regularly; Google, for example, generally refreshes its copy within 24 hours. Being cited again after unblocking takes longer, because the assistant has to crawl your pages again before it can quote them.

Does blocking AI crawlers affect Google rankings?

No. Blocking GPTBot, ClaudeBot, PerplexityBot, Google-Extended or any other AI crawler has no effect on Google Search rankings. Only rules that apply to Googlebot change how Google crawls your site.

What to do next

1. Run your site through the AI crawler checker and note which assistants can cite you.
2. Decide what AI may do on your site, using the table above.
3. Copy the matching robots.txt group, publish it, and check again the next day.
4. Being readable is the first step. To be quoted, pages need clear answers, authorship and structure: see how to appear in AI search, and add an llms.txt file that points AI tools to your best pages.