robots.txt rules
Copy the group you need into the robots.txt file at the root of your site. Keep only one option per bot.
At a glance
| Detail | Applebot |
|---|---|
| Operator | Apple |
| Type | AI search crawler |
| What it does | Crawls pages for Siri, Spotlight and Safari search features. |
| robots.txt token | Applebot |
| Example user agent | Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot) |
| Follows robots.txt | Apple says Applebot follows robots.txt. If there are no rules for Applebot, it follows the rules for Googlebot. |
| How to verify | Use a reverse DNS lookup that resolves to applebot.apple.com. |
| If you block it | Your pages may stop appearing in Siri and Spotlight suggestions. |
| Source | Apple support article about Applebot |
User agent version numbers change over time. Match on the token, not the full string.
What Applebot does
Applebot powers search features across Apple products, such as Siri and Spotlight suggestions. Apple also uses a separate token, Applebot-Extended, to let sites opt out of having that content used to train Apple's AI models.
Should you allow or block it?
Allow it. Use Applebot-Extended separately if you want to opt out of Apple AI training.
If you block it: Your pages may stop appearing in Siri and Spotlight suggestions.
- Training crawlers decide whether your content helps build future models. Blocking them does not remove you from AI search today.
- Search crawlers decide whether AI engines can find and cite your pages. Blocking them usually costs visibility.
- User triggered fetchers read a page because a person asked. Blocking them stops the AI reading you at the moment someone is interested.
How to find Applebot in your logs
Search your server or CDN access logs for the token Applebot. This command counts visits by IP, path and status code in a standard access log:
grep -i "Applebot" access.log | awk '{print $1, $7, $9}' | sort | uniq -c | sort -rn | head
A run of 403 or 503 status codes means your firewall or bot protection is blocking it, even if robots.txt allows it. That is one of the most common hidden reasons sites lose AI visibility.
How to tell if a visit is genuine
Anyone can put Applebot in a user agent string, so the name alone proves nothing. Use a reverse DNS lookup that resolves to applebot.apple.com.
Other Apple bots
- Applebot-Extended: control token (does not crawl). A robots.txt token that controls whether Apple may use Applebot crawled content to train its AI models.