robots.txt rules
Copy the group you need into the robots.txt file at the root of your site. Keep only one option per bot.
At a glance
| Detail | Google-Extended |
|---|---|
| Operator | |
| Type | Control token (does not crawl) |
| What it does | A robots.txt token that controls whether Google may use your content for Gemini model training and grounding. |
| robots.txt token | Google-Extended |
| Example user agent | None. It does not send requests. |
| Follows robots.txt | It exists only as a robots.txt token. There is no Google-Extended user agent in your logs. |
| How to verify | Nothing to verify. Any request claiming to be Google-Extended is not from Google. |
| If you block it | Google will not use your content for Gemini training or Gemini app grounding. Google Search, AI Overviews and AI Mode are not affected. |
| Source | Google Search Central list of common crawlers |
User agent version numbers change over time. Match on the token, not the full string.
What Google-Extended does
Google-Extended never visits your site. Crawling is done by Google's normal crawlers, and this token tells Google whether that content can be used to train future Gemini models and to ground answers in the Gemini apps and on Vertex AI. Google says it does not affect inclusion or ranking in Google Search.
Should you allow or block it?
Your choice. It is a training and grounding decision, separate from Search visibility.
If you block it: Google will not use your content for Gemini training or Gemini app grounding. Google Search, AI Overviews and AI Mode are not affected.
- Training crawlers decide whether your content helps build future models. Blocking them does not remove you from AI search today.
- Search crawlers decide whether AI engines can find and cite your pages. Blocking them usually costs visibility.
- User triggered fetchers read a page because a person asked. Blocking them stops the AI reading you at the moment someone is interested.
How to tell if a visit is genuine
Anyone can put Google-Extended in a user agent string, so the name alone proves nothing. Nothing to verify. Any request claiming to be Google-Extended is not from Google.
Other Google bots
- Googlebot: ai search crawler. Crawls pages for Google Search, which also feeds AI Overviews and AI Mode.
- Google-CloudVertexBot: ai search crawler. Crawls sites when their owners ask, to build Vertex AI agents.