robots.txt rules
Copy the group you need into the robots.txt file at the root of your site. Keep only one option per bot.
At a glance
| Detail | Amazonbot |
|---|---|
| Operator | Amazon |
| Type | Training crawler |
| What it does | Crawls pages to improve Amazon services such as Alexa's question answering. |
| robots.txt token | Amazonbot |
| Example user agent | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) |
| Follows robots.txt | Amazon says Amazonbot follows robots.txt. |
| How to verify | Use a reverse DNS lookup that resolves to crawl.amazonbot.amazon. |
| If you block it | Amazon services, including Alexa, may have less information from your site. |
| Source | Amazon developer documentation for Amazonbot |
User agent version numbers change over time. Match on the token, not the full string.
What Amazonbot does
Amazonbot is Amazon's web crawler. Amazon says it is used to improve its products and services, including helping Alexa answer more questions accurately. Its content may also support Amazon's AI features.
Should you allow or block it?
Your choice. Allow it if Alexa or Amazon's AI features matter to your audience.
If you block it: Amazon services, including Alexa, may have less information from your site.
- Training crawlers decide whether your content helps build future models. Blocking them does not remove you from AI search today.
- Search crawlers decide whether AI engines can find and cite your pages. Blocking them usually costs visibility.
- User triggered fetchers read a page because a person asked. Blocking them stops the AI reading you at the moment someone is interested.
How to find Amazonbot in your logs
Search your server or CDN access logs for the token Amazonbot. This command counts visits by IP, path and status code in a standard access log:
grep -i "Amazonbot" access.log | awk '{print $1, $7, $9}' | sort | uniq -c | sort -rn | head
A run of 403 or 503 status codes means your firewall or bot protection is blocking it, even if robots.txt allows it. That is one of the most common hidden reasons sites lose AI visibility.
How to tell if a visit is genuine
Anyone can put Amazonbot in a user agent string, so the name alone proves nothing. Use a reverse DNS lookup that resolves to crawl.amazonbot.amazon.