Can ChatGPT see your website?
Enter your site to see which AI crawlers your robots.txt lets in: GPTBot, ClaudeBot, PerplexityBot, Google-Extended and 11 more. Find out which assistants can cite you, and the exact line to change.
- Free, no sign-up
- 15 crawlers checked
- Results in seconds
- Ready-to-copy fixes
Try it: openai.com · nytimes.com · wikipedia.org
Being cited and being trained on are different switches
Most sites that block AI block the wrong crawlers. Each AI company runs separate crawlers for answering questions and for training models.
AI search crawlers
OAI-SearchBot, Claude-SearchBot, PerplexityBot and the user agents fetch your page when someone asks a question, so the assistant can quote and link to you. Block these and you disappear from AI answers.
AI training crawlers
GPTBot, ClaudeBot, Google-Extended, CCBot and others collect content to train future models. Blocking these opts you out of training but does not stop assistants citing you today.
Search engines
Googlebot feeds Google Search, AI Overviews and AI Mode. Bingbot feeds Bing and Microsoft Copilot. You cannot opt out of AI Overviews without leaving Google Search.
How Indian websites treat AI crawlers
We ran this check on 80 well-known Indian websites, 20 each from news, edtech, universities and government, on 21 October 2026.
Most news publishers that block GPTBot also block OAI-SearchBot and ChatGPT-User, so they opt out of being cited in ChatGPT, not just of training. Education websites are almost entirely open. Sites whose robots.txt could not be read are left out of each count.
Every AI crawler the checker looks for
The user-agent names to use in robots.txt, who runs each crawler and what it does.
| User-agent | Company | Kind | What it does |
|---|---|---|---|
OAI-SearchBot | OpenAI | AI search and answers | Finds and cites pages in ChatGPT search answers. |
ChatGPT-User | OpenAI | AI search and answers | Opens a page when a ChatGPT user asks about it or shares the link. |
GPTBot | OpenAI | AI training | Collects content to train OpenAI's models. |
Claude-SearchBot | Anthropic | AI search and answers | Finds pages to cite in Claude's search answers. |
Claude-User | Anthropic | AI search and answers | Opens a page when a Claude user asks about it. |
ClaudeBot | Anthropic | AI training | Collects content to train Anthropic's Claude models. |
PerplexityBot | Perplexity | AI search and answers | Finds and cites pages in Perplexity answers. |
Perplexity-User | Perplexity | AI search and answers | Opens a page when a Perplexity user asks about it. |
Googlebot | AI search and answers | Google Search, including AI Overviews and AI Mode. | |
Google-Extended | AI training | Lets Google use content for Gemini training and grounding. Does not affect Search or AI Overviews. | |
Bingbot | Microsoft | AI search and answers | Bing search, which Microsoft Copilot and other assistants answer from. |
Applebot-Extended | Apple | AI training | Lets Apple use content to train Apple Intelligence models. |
meta-externalagent | Meta | AI training | Collects content to train Meta's Llama models. |
CCBot | Common Crawl | AI training | Builds the open web archive that many AI models are trained on. |
Bytespider | ByteDance | AI training | Collects content for ByteDance's AI models. |
How do you allow or block an AI crawler?
Add a group to the robots.txt file at the root of your site. Name the crawler on a User-agent line, then say Allow: / or
Disallow: /. Several crawlers can share one group. The checker gives you ready-made groups for the three usual choices, and our
robots.txt guide explains the rules Google follows. For every AI crawler, what it controls and which to block,
read our complete guide to AI crawlers.
Why does my site block AI crawlers when nobody chose to?
- A security plugin or CDN setting. Some turn on AI bot blocking with one click, or by default.
- A copied block list. Lists shared online often mix search and training crawlers.
- A wildcard rule.
User-agent: *withDisallow: /blocks every crawler without a group of its own, AI ones included. - No readable file. If robots.txt returns a server error, crawlers treat the whole site as blocked until it can be read.
Want to know whether assistants actually mention you? Track your brand in ChatGPT, Perplexity and Gemini, or run a free SEO check of any page.
Questions about AI crawlers
How do I check if my site blocks GPTBot?
Enter your website above. The checker reads your robots.txt the way the crawlers do and shows, for GPTBot and every other AI crawler, whether it is allowed or blocked and the exact line that decides it. You can also open yoursite.com/robots.txt and look for "User-agent: GPTBot" followed by "Disallow: /".
Does blocking GPTBot remove my site from ChatGPT?
No. GPTBot only collects content for training OpenAI's models. ChatGPT finds and cites pages with OAI-SearchBot, and opens links for users with ChatGPT-User. Block GPTBot alone and ChatGPT can still cite you; block OAI-SearchBot and it cannot.
Does blocking Google-Extended remove me from AI Overviews?
No. Google-Extended only controls whether Google may use your content for Gemini training and grounding. Google Search, AI Overviews and AI Mode use the normal Googlebot, so the only way to leave AI Overviews is to leave Google Search.
Should I block AI crawlers?
It depends on what your content is for. If people need to find your business, allow the AI search crawlers so assistants can cite you; you can still block the training crawlers. If your content is the product, such as news or paid courses, blocking training crawlers protects it, but blocking the search crawlers too makes you invisible in AI answers.
Is robots.txt enough to block AI crawlers?
The major AI companies say their crawlers obey robots.txt, but it is a request, not a lock. To enforce a block, use your CDN or firewall's bot rules as well. The checker's firewall test shows whether your server already refuses requests that carry AI crawler names.
What does the firewall test do?
It asks for your page once as a normal browser and once under each AI crawler's name. If the page opens for the browser but not for a crawler, a firewall or CDN rule is probably turning that crawler away before it reads robots.txt. It is indicative: the real crawlers come from their companies' own addresses, which some firewalls check.
Is the checker free?
Yes. No sign-up and no cost. It only reads your public robots.txt and asks for one page, the same as any visitor.