What is robots.txt?

By the Variask team · Updated

Definition

robots.txt is a text file at the root of a website that tells crawlers, including AI crawlers, which parts of the site they may or may not visit.

With Variask: The free Variask robots.txt checker and AI crawler checker show which AI bots can reach your site.

How it works for AI crawlers

Each rule names a user agent, such as GPTBot or PerplexityBot, and the paths it may not crawl. Reputable crawlers read the file before crawling. It is a request, not a lock, and it does not stop bots that ignore it.

What it means for brands

  • Review robots.txt after site changes; a single wrong line can hide your whole site from AI search.
  • Remember that firewalls and bot protection can block crawlers that robots.txt allows.

Questions

robots.txt:
common questions

It stops future crawling by bots that respect it. It does not remove content already collected.

Related terms

Keep exploring

Technical

AI crawlers

AI crawlers are bots run by AI companies that visit websites to collect content, either to train models or to fetch pages live when an assistant answers a question.

Read

Technical

GPTBot

GPTBot is OpenAI's web crawler that collects publicly available pages that may be used to train OpenAI's models. It is separate from OAI-SearchBot, which powers search in ChatGPT.

Read

Technical

llms.txt

llms.txt is a proposed standard: a markdown file at the root of a website that gives AI models a short summary of the site and links to its most useful pages.

Read
Full glossary