robots.txt rules
Copy the group you need into the robots.txt file at the root of your site. Keep only one option per bot.
At a glance
| Detail | Applebot-Extended |
|---|---|
| Operator | Apple |
| Type | Control token (does not crawl) |
| What it does | A robots.txt token that controls whether Apple may use Applebot crawled content to train its AI models. |
| robots.txt token | Applebot-Extended |
| Example user agent | None. It does not send requests. |
| Follows robots.txt | It exists only as a robots.txt token. |
| How to verify | Nothing to verify. Any request claiming to be Applebot-Extended is not from Apple. |
| If you block it | Your content is not used to train Apple's foundation models. Applebot search features are not affected. |
| Source | Apple support article about Applebot |
User agent version numbers change over time. Match on the token, not the full string.
What Applebot-Extended does
Apple says Applebot-Extended does not crawl webpages. It is a token in robots.txt that tells Apple whether content Applebot has already crawled can be used to train Apple's foundation models for Apple Intelligence.
Should you allow or block it?
Your choice. It is a training decision only.
If you block it: Your content is not used to train Apple's foundation models. Applebot search features are not affected.
- Training crawlers decide whether your content helps build future models. Blocking them does not remove you from AI search today.
- Search crawlers decide whether AI engines can find and cite your pages. Blocking them usually costs visibility.
- User triggered fetchers read a page because a person asked. Blocking them stops the AI reading you at the moment someone is interested.
How to tell if a visit is genuine
Anyone can put Applebot-Extended in a user agent string, so the name alone proves nothing. Nothing to verify. Any request claiming to be Applebot-Extended is not from Apple.
Other Apple bots
- Applebot: ai search crawler. Crawls pages for Siri, Spotlight and Safari search features.