Key facts
- Blocking a training crawler does not remove you from that company's AI search results.
- Blocking a search crawler such as OAI-SearchBot can keep your pages out of answers that cite live sources.
- Google-Extended controls Gemini training and grounding use; it does not affect Google Search or AI Overviews.
- Firewalls and bot protection can block AI crawlers even when robots.txt allows them.
The main AI crawlers
- OpenAI: GPTBot (training), OAI-SearchBot (ChatGPT search index), ChatGPT-User (pages fetched for a user).
- Anthropic: ClaudeBot (training), Claude-SearchBot (search), Claude-User (pages fetched for a user).
- Perplexity: PerplexityBot (search index), Perplexity-User (pages fetched for a user).
- Google: Googlebot (Search and AI Overviews), Google-Extended (a control token for Gemini training and grounding).
- Others: Applebot and Applebot-Extended, Bingbot (which also feeds Copilot), Meta's meta-externalagent, Amazonbot and CCBot (Common Crawl).
A sensible default
Most brands that want to be recommended should allow the search and user fetch crawlers, since those decide whether your pages can be read and cited in live answers. Allowing training crawlers helps future models know your brand; blocking them is a business choice about how your content is used. Whatever you choose, check your firewall and CDN bot settings too, because they often block AI bots silently.
How to check your site
- 1Read your robots.txt and look for groups that name each crawler.
- 2Remember that a crawler follows only the most specific group that names it, falling back to *.
- 3Request a page with each crawler's user agent to see whether your firewall lets it through.
- 4Watch your server logs or analytics for visits from these bots after you change the rules.
Put this into practice with Variask AI traffic and crawlers, Variask Site health and the free AI crawler access checker. See Variask pricing or read the Variask FAQ.