robots.txt rules
Copy the group you need into the robots.txt file at the root of your site. Keep only one option per bot.
At a glance
| Detail | meta-externalagent |
|---|---|
| Operator | Meta |
| Type | Training crawler |
| What it does | Crawls pages for training Meta's AI models and for indexing content directly in Meta products. |
| robots.txt token | meta-externalagent |
| Example user agent | meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler) |
| Follows robots.txt | Meta says it follows robots.txt, and that it can cache robots.txt for up to 24 hours. |
| How to verify | Match the user agent and check behaviour in your logs against Meta's crawler documentation. |
| If you block it | Meta stops crawling your content for AI training and related product indexing. |
| Source | Meta for Developers crawler documentation |
User agent version numbers change over time. Match on the token, not the full string.
What meta-externalagent does
Meta-ExternalAgent is Meta's crawler for uses such as training AI models or improving products by indexing content directly. It is the main Meta token to manage if you want to control AI use of your content.
Should you allow or block it?
Your choice. Decide based on how you feel about AI training use.
If you block it: Meta stops crawling your content for AI training and related product indexing.
- Training crawlers decide whether your content helps build future models. Blocking them does not remove you from AI search today.
- Search crawlers decide whether AI engines can find and cite your pages. Blocking them usually costs visibility.
- User triggered fetchers read a page because a person asked. Blocking them stops the AI reading you at the moment someone is interested.
How to find meta-externalagent in your logs
Search your server or CDN access logs for the token meta-externalagent. This command counts visits by IP, path and status code in a standard access log:
grep -i "meta-externalagent" access.log | awk '{print $1, $7, $9}' | sort | uniq -c | sort -rn | head
A run of 403 or 503 status codes means your firewall or bot protection is blocking it, even if robots.txt allows it. That is one of the most common hidden reasons sites lose AI visibility.
How to tell if a visit is genuine
Anyone can put meta-externalagent in a user agent string, so the name alone proves nothing. Match the user agent and check behaviour in your logs against Meta's crawler documentation.
Other Meta bots
- meta-externalfetcher: user triggered fetcher. Fetches individual links when a Meta AI user asks for them.