What is a user agent?

By the Variask team · Updated

Definition

A user agent is the name a browser or crawler sends with each request, such as GPTBot or Googlebot. Websites and robots.txt rules use it to tell AI crawlers apart and decide what each may access.

Also called User agent string

How it works

  • Every request includes a User-Agent header.
  • Crawlers publish their user agent names.
  • robots.txt groups rules by user agent.
  • Firewalls can allow or block by user agent and IP address.

What it means for brands

  • Know the user agents of the AI crawlers you want to allow.
  • User agents can be faked, so serious bot rules also check IP addresses.

Questions

User agent:
common questions

OpenAI uses OAI-SearchBot for search, ChatGPT-User for user requests and GPTBot for training.

Check your server logs, or paste them into the free AI bot log analyzer.

Related terms

Keep exploring

Technical

AI crawlers

AI crawlers are bots run by AI companies that visit websites to collect content, either to train models or to fetch pages live when an assistant answers a question.

Read

Technical

GPTBot

GPTBot is OpenAI's web crawler that collects publicly available pages that may be used to train OpenAI's models. It is separate from OAI-SearchBot, which powers search in ChatGPT.

Read

Technical

robots.txt

robots.txt is a text file at the root of a website that tells crawlers, including AI crawlers, which parts of the site they may or may not visit.

Read
Full glossary