Every AI crawler, explained

GPTBot

GPTBot: what it is and how to allow or block it

GPTBot is OpenAI's training crawler. Collects public web pages that may be used to train OpenAI's models. See its user agent, robots.txt rules and whether to allow or block it.

OAI-SearchBot

OAI-SearchBot: what it is and how to allow or block it

OAI-SearchBot is OpenAI's ai search crawler. Finds and indexes pages so they can appear and be cited in ChatGPT search. See its user agent, robots.txt rules and whether to allow or block it.

ChatGPT-User

ChatGPT-User: what it is and how to allow or block it

ChatGPT-User is OpenAI's user triggered fetcher. Opens a page when a ChatGPT user, custom GPT or agent asks it to. See its user agent, robots.txt rules and whether to allow or block it.

ClaudeBot

ClaudeBot: what it is and how to allow or block it

ClaudeBot is Anthropic's training crawler. Collects public web content that may be used to train Anthropic's Claude models. See its user agent, robots.txt rules and whether to allow or block it.

Claude-SearchBot

Claude-SearchBot: what it is and how to allow or block it

Claude-SearchBot is Anthropic's ai search crawler. Crawls the web to improve the quality and relevance of Claude's search results. See its user agent, robots.txt rules and whether to allow or block it.

Claude-User

Claude-User: what it is and how to allow or block it

Claude-User is Anthropic's user triggered fetcher. Fetches a page when a Claude user asks a question that needs it. See its user agent, robots.txt rules and whether to allow or block it.

PerplexityBot

PerplexityBot: what it is and how to allow or block it

PerplexityBot is Perplexity's ai search crawler. Indexes pages so they can appear as sources in Perplexity answers. See its user agent, robots.txt rules and whether to allow or block it.

Perplexity-User

Perplexity-User: what it is and how to allow or block it

Perplexity-User is Perplexity's user triggered fetcher. Visits a page when a Perplexity user asks a question that needs it. See its user agent, robots.txt rules and whether to allow or block it.

Googlebot

Googlebot: what it is and how to allow or block it

Googlebot is Google's ai search crawler. Crawls pages for Google Search, which also feeds AI Overviews and AI Mode. See its user agent, robots.txt rules and whether to allow or block it.

Google-Extended

Google-Extended: what it is and how to allow or block it

Google-Extended is Google's control token (does not crawl). A robots.txt token that controls whether Google may use your content for Gemini model training and grounding. See its user agent, robots.txt rules and whether to allow or block it.

Google-CloudVertexBot

Google-CloudVertexBot: what it is and how to allow or block it

Google-CloudVertexBot is Google's ai search crawler. Crawls sites when their owners ask, to build Vertex AI agents. See its user agent, robots.txt rules and whether to allow or block it.

Bingbot

Bingbot: what it is and how to allow or block it

Bingbot is Microsoft's ai search crawler. Crawls pages for Bing, which grounds many Microsoft Copilot answers. See its user agent, robots.txt rules and whether to allow or block it.

Applebot

Applebot: what it is and how to allow or block it

Applebot is Apple's ai search crawler. Crawls pages for Siri, Spotlight and Safari search features. See its user agent, robots.txt rules and whether to allow or block it.

Applebot-Extended

Applebot-Extended: what it is and how to allow or block it

Applebot-Extended is Apple's control token (does not crawl). A robots.txt token that controls whether Apple may use Applebot crawled content to train its AI models. See its user agent, robots.txt rules and whether to allow or block it.

meta-externalagent

meta-externalagent: what it is and how to allow or block it

meta-externalagent is Meta's training crawler. Crawls pages for training Meta's AI models and for indexing content directly in Meta products. See its user agent, robots.txt rules and whether to allow or block it.

meta-externalfetcher

meta-externalfetcher: what it is and how to allow or block it

meta-externalfetcher is Meta's user triggered fetcher. Fetches individual links when a Meta AI user asks for them. See its user agent, robots.txt rules and whether to allow or block it.

Amazonbot

Amazonbot: what it is and how to allow or block it

Amazonbot is Amazon's training crawler. Crawls pages to improve Amazon services such as Alexa's question answering. See its user agent, robots.txt rules and whether to allow or block it.

Bytespider

Bytespider: what it is and how to allow or block it

Bytespider is ByteDance's training crawler. A ByteDance crawler widely linked to data collection for its AI products. See its user agent, robots.txt rules and whether to allow or block it.

CCBot

CCBot: what it is and how to allow or block it

CCBot is Common Crawl's training crawler. Builds Common Crawl's open web dataset, which many AI models are trained on. See its user agent, robots.txt rules and whether to allow or block it.

DuckAssistBot

DuckAssistBot: what it is and how to allow or block it

DuckAssistBot is DuckDuckGo's user triggered fetcher. Fetches pages for DuckDuckGo's AI assisted answers. See its user agent, robots.txt rules and whether to allow or block it.

MistralAI-User

MistralAI-User: what it is and how to allow or block it

MistralAI-User is Mistral AI's user triggered fetcher. Fetches pages when users of Mistral's Le Chat ask for current web content. See its user agent, robots.txt rules and whether to allow or block it.