robots.txt rules
Copy the group you need into the robots.txt file at the root of your site. Keep only one option per bot.
At a glance
| Detail | Googlebot |
|---|---|
| Operator | |
| Type | AI search crawler |
| What it does | Crawls pages for Google Search, which also feeds AI Overviews and AI Mode. |
| robots.txt token | Googlebot |
| Example user agent | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) |
| Follows robots.txt | Google says its common crawlers always follow robots.txt when crawling automatically. |
| How to verify | Use a reverse DNS lookup that resolves to googlebot.com or google.com, or check Google's published IP ranges. |
| If you block it | Your site drops out of Google Search and its AI features. To limit how text is shown in AI Overviews, Google points to snippet controls such as nosnippet instead. |
| Source | Google Search Central crawler documentation |
User agent version numbers change over time. Match on the token, not the full string.
What Googlebot does
Googlebot is Google's main search crawler. There is no separate crawler for AI Overviews or AI Mode: they draw on pages Googlebot has indexed. Blocking Googlebot removes you from Google Search entirely, including its AI features.
Should you allow or block it?
Always allow it unless you want to leave Google Search.
If you block it: Your site drops out of Google Search and its AI features. To limit how text is shown in AI Overviews, Google points to snippet controls such as nosnippet instead.
- Training crawlers decide whether your content helps build future models. Blocking them does not remove you from AI search today.
- Search crawlers decide whether AI engines can find and cite your pages. Blocking them usually costs visibility.
- User triggered fetchers read a page because a person asked. Blocking them stops the AI reading you at the moment someone is interested.
How to find Googlebot in your logs
Search your server or CDN access logs for the token Googlebot. This command counts visits by IP, path and status code in a standard access log:
grep -i "Googlebot" access.log | awk '{print $1, $7, $9}' | sort | uniq -c | sort -rn | head
A run of 403 or 503 status codes means your firewall or bot protection is blocking it, even if robots.txt allows it. That is one of the most common hidden reasons sites lose AI visibility.
How to tell if a visit is genuine
Anyone can put Googlebot in a user agent string, so the name alone proves nothing. Use a reverse DNS lookup that resolves to googlebot.com or google.com, or check Google's published IP ranges.
Other Google bots
- Google-Extended: control token (does not crawl). A robots.txt token that controls whether Google may use your content for Gemini model training and grounding.
- Google-CloudVertexBot: ai search crawler. Crawls sites when their owners ask, to build Vertex AI agents.