robots.txt rules
Copy the group you need into the robots.txt file at the root of your site. Keep only one option per bot.
At a glance
| Detail | Google-CloudVertexBot |
|---|---|
| Operator | |
| Type | AI search crawler |
| What it does | Crawls sites when their owners ask, to build Vertex AI agents. |
| robots.txt token | Google-CloudVertexBot |
| Example user agent | Contains "Google-CloudVertexBot" |
| Follows robots.txt | Google says its common crawlers follow robots.txt when crawling automatically. |
| How to verify | Use reverse DNS or Google's published IP ranges for its common crawlers. |
| If you block it | Vertex AI agents built on your site cannot crawl it. Google Search is not affected. |
| Source | Google Search Central list of common crawlers |
User agent version numbers change over time. Match on the token, not the full string.
What Google-CloudVertexBot does
Google-CloudVertexBot crawls pages for Vertex AI Agents, usually when a business sets up an agent that uses its own website. Google says rules for this token have no effect on Google Search or other products.
Should you allow or block it?
Allow it if you or a partner build AI agents on your content. Otherwise it rarely matters.
If you block it: Vertex AI agents built on your site cannot crawl it. Google Search is not affected.
- Training crawlers decide whether your content helps build future models. Blocking them does not remove you from AI search today.
- Search crawlers decide whether AI engines can find and cite your pages. Blocking them usually costs visibility.
- User triggered fetchers read a page because a person asked. Blocking them stops the AI reading you at the moment someone is interested.
How to find Google-CloudVertexBot in your logs
Search your server or CDN access logs for the token Google-CloudVertexBot. This command counts visits by IP, path and status code in a standard access log:
grep -i "Google-CloudVertexBot" access.log | awk '{print $1, $7, $9}' | sort | uniq -c | sort -rn | head
A run of 403 or 503 status codes means your firewall or bot protection is blocking it, even if robots.txt allows it. That is one of the most common hidden reasons sites lose AI visibility.
How to tell if a visit is genuine
Anyone can put Google-CloudVertexBot in a user agent string, so the name alone proves nothing. Use reverse DNS or Google's published IP ranges for its common crawlers.
Other Google bots
- Googlebot: ai search crawler. Crawls pages for Google Search, which also feeds AI Overviews and AI Mode.
- Google-Extended: control token (does not crawl). A robots.txt token that controls whether Google may use your content for Gemini model training and grounding.