robots.txt rules
Copy the group you need into the robots.txt file at the root of your site. Keep only one option per bot.
At a glance
| Detail | CCBot |
|---|---|
| Operator | Common Crawl |
| Type | Training crawler |
| What it does | Builds Common Crawl's open web dataset, which many AI models are trained on. |
| robots.txt token | CCBot |
| Example user agent | CCBot/2.0 (https://commoncrawl.org/faq/) |
| Follows robots.txt | Common Crawl says CCBot follows robots.txt. |
| How to verify | Match the user agent and check Common Crawl's published guidance. |
| If you block it | Your future content is left out of the Common Crawl archive, and so out of datasets built from it. |
| Source | Common Crawl FAQ |
User agent version numbers change over time. Match on the token, not the full string.
What CCBot does
CCBot is run by Common Crawl, a non profit that publishes a free, open archive of the web. Researchers and companies use this dataset widely, including to train language models. Blocking CCBot keeps your future content out of that archive.
Should you allow or block it?
Your choice. Blocking it is a broad training decision, since many models use this dataset.
If you block it: Your future content is left out of the Common Crawl archive, and so out of datasets built from it.
- Training crawlers decide whether your content helps build future models. Blocking them does not remove you from AI search today.
- Search crawlers decide whether AI engines can find and cite your pages. Blocking them usually costs visibility.
- User triggered fetchers read a page because a person asked. Blocking them stops the AI reading you at the moment someone is interested.
How to find CCBot in your logs
Search your server or CDN access logs for the token CCBot. This command counts visits by IP, path and status code in a standard access log:
grep -i "CCBot" access.log | awk '{print $1, $7, $9}' | sort | uniq -c | sort -rn | head
A run of 403 or 503 status codes means your firewall or bot protection is blocking it, even if robots.txt allows it. That is one of the most common hidden reasons sites lose AI visibility.
How to tell if a visit is genuine
Anyone can put CCBot in a user agent string, so the name alone proves nothing. Match the user agent and check Common Crawl's published guidance.