robots.txt rules
Copy the group you need into the robots.txt file at the root of your site. Keep only one option per bot.
At a glance
| Detail | PerplexityBot |
|---|---|
| Operator | Perplexity |
| Type | AI search crawler |
| What it does | Indexes pages so they can appear as sources in Perplexity answers. |
| robots.txt token | PerplexityBot |
| Example user agent | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot) |
| Follows robots.txt | Perplexity says PerplexityBot follows robots.txt, and that changes may take up to 24 hours to apply. |
| How to verify | Check the request IP against perplexity.com/perplexitybot.json. |
| If you block it | Your pages can stop appearing as cited sources in Perplexity. |
| Source | Perplexity crawler documentation (docs.perplexity.ai) |
User agent version numbers change over time. Match on the token, not the full string.
What PerplexityBot does
PerplexityBot builds Perplexity's search index. Perplexity says it is used to surface and link websites in its results, not to train foundation models. Perplexity cites sources on almost every answer, so being indexed matters.
Should you allow or block it?
Allow it if you want Perplexity to cite you.
If you block it: Your pages can stop appearing as cited sources in Perplexity.
- Training crawlers decide whether your content helps build future models. Blocking them does not remove you from AI search today.
- Search crawlers decide whether AI engines can find and cite your pages. Blocking them usually costs visibility.
- User triggered fetchers read a page because a person asked. Blocking them stops the AI reading you at the moment someone is interested.
How to find PerplexityBot in your logs
Search your server or CDN access logs for the token PerplexityBot. This command counts visits by IP, path and status code in a standard access log:
grep -i "PerplexityBot" access.log | awk '{print $1, $7, $9}' | sort | uniq -c | sort -rn | head
A run of 403 or 503 status codes means your firewall or bot protection is blocking it, even if robots.txt allows it. That is one of the most common hidden reasons sites lose AI visibility.
How to tell if a visit is genuine
Anyone can put PerplexityBot in a user agent string, so the name alone proves nothing. Check the request IP against perplexity.com/perplexitybot.json.
Other Perplexity bots
- Perplexity-User: user triggered fetcher. Visits a page when a Perplexity user asks a question that needs it.