robots.txt rules
Copy the group you need into the robots.txt file at the root of your site. Keep only one option per bot.
At a glance
| Detail | ClaudeBot |
|---|---|
| Operator | Anthropic |
| Type | Training crawler |
| What it does | Collects public web content that may be used to train Anthropic's Claude models. |
| robots.txt token | ClaudeBot |
| Example user agent | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com) |
| Follows robots.txt | Anthropic says its bots follow robots.txt, including Disallow rules and the non standard Crawl-delay rule. |
| How to verify | Match the user agent and watch behaviour in your logs. Anthropic's documentation centres on robots.txt controls. |
| If you block it | Your future content is excluded from Anthropic's training data. Claude search and user fetches are not affected. |
| Source | Anthropic help center article on crawlers and robots.txt |
User agent version numbers change over time. Match on the token, not the full string.
What ClaudeBot does
ClaudeBot is Anthropic's training crawler. Anthropic says blocking it excludes your site's future content from its training data. It does not control Claude search or the fetches Claude makes for users, which have their own tokens.
Should you allow or block it?
Your choice. If crawl volume is the concern, Crawl-delay lets you slow it down instead of blocking it.
If you block it: Your future content is excluded from Anthropic's training data. Claude search and user fetches are not affected.
- Training crawlers decide whether your content helps build future models. Blocking them does not remove you from AI search today.
- Search crawlers decide whether AI engines can find and cite your pages. Blocking them usually costs visibility.
- User triggered fetchers read a page because a person asked. Blocking them stops the AI reading you at the moment someone is interested.
How to find ClaudeBot in your logs
Search your server or CDN access logs for the token ClaudeBot. This command counts visits by IP, path and status code in a standard access log:
grep -i "ClaudeBot" access.log | awk '{print $1, $7, $9}' | sort | uniq -c | sort -rn | head
A run of 403 or 503 status codes means your firewall or bot protection is blocking it, even if robots.txt allows it. That is one of the most common hidden reasons sites lose AI visibility.
How to tell if a visit is genuine
Anyone can put ClaudeBot in a user agent string, so the name alone proves nothing. Match the user agent and watch behaviour in your logs. Anthropic's documentation centres on robots.txt controls.
Other Anthropic bots
- Claude-SearchBot: ai search crawler. Crawls the web to improve the quality and relevance of Claude's search results.
- Claude-User: user triggered fetcher. Fetches a page when a Claude user asks a question that needs it.