What are AI crawlers?

By the Variask team · Updated

Definition

AI crawlers are bots run by AI companies that visit websites to collect content, either to train models or to fetch pages live when an assistant answers a question.

Also called AI bots, LLM crawlers

With Variask: Variask checks crawler access for every major AI bot and shows which AI crawlers visit your site and how often.

The main types of AI crawler

  • Training crawlers collect content to train future models, such as GPTBot from OpenAI, ClaudeBot from Anthropic and CCBot from Common Crawl.
  • Search crawlers build the index an assistant searches, such as OAI-SearchBot, Claude-SearchBot and PerplexityBot.
  • User triggered fetchers visit a page because a person asked the assistant to, such as ChatGPT-User, Claude-User and Perplexity-User.
  • Control tokens such as Google-Extended and Applebot-Extended let you opt out of AI training use without blocking normal search crawling.

How to manage them

  1. 1Decide per bot: many brands allow search and user crawlers so they can be cited, and choose separately on training crawlers.
  2. 2Set rules in robots.txt by user agent.
  3. 3Check your firewall, CDN and bot protection; they often block AI crawlers even when robots.txt allows them.
  4. 4Watch your server logs to see which bots visit and which pages they read.

What it means for brands

  • Blocking search crawlers can keep you out of AI answers and citations entirely.
  • Most AI crawlers do not run JavaScript, so content must be in the HTML.
Deep dive in Variask UniversityAI crawlers explained: GPTBot, OAI-SearchBot, ClaudeBot and more

Questions

AI crawlers:
common questions

Blocking GPTBot stops OpenAI training use but does not remove you from ChatGPT search, which uses OAI-SearchBot. Decide on each bot separately.

The major AI companies say their crawlers respect robots.txt. User triggered fetchers may behave differently; check each provider's documentation.

Related terms

Keep exploring

Technical

GPTBot

GPTBot is OpenAI's web crawler that collects publicly available pages that may be used to train OpenAI's models. It is separate from OAI-SearchBot, which powers search in ChatGPT.

Read

Technical

robots.txt

robots.txt is a text file at the root of a website that tells crawlers, including AI crawlers, which parts of the site they may or may not visit.

Read

Technical

Server-side rendering (SSR)

Server-side rendering (SSR) means a web page's HTML is built on the server and sent complete to the browser or bot, instead of being assembled by JavaScript in the browser. It makes content readable to crawlers that do not run JavaScript.

Read

Technical

llms.txt

llms.txt is a proposed standard: a markdown file at the root of a website that gives AI models a short summary of the site and links to its most useful pages.

Read

Measurement

AI referral traffic

AI referral traffic is the visits your website gets from people clicking links in AI assistants such as ChatGPT, Perplexity, Gemini, Claude and Copilot.

Read
Full glossary