robots.txt rules
Copy the group you need into the robots.txt file at the root of your site. Keep only one option per bot.
At a glance
| Detail | ChatGPT-User |
|---|---|
| Operator | OpenAI |
| Type | User triggered fetcher |
| What it does | Opens a page when a ChatGPT user, custom GPT or agent asks it to. |
| robots.txt token | ChatGPT-User |
| Example user agent | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot |
| Follows robots.txt | Because a user starts each request, OpenAI says robots.txt rules may not apply to it in the same way as to its automatic crawlers. |
| How to verify | Check the request IP against openai.com/chatgpt-user.json. |
| If you block it | ChatGPT may be unable to read your page when a user asks about it, so answers can be thinner or rely on other sources. |
| Source | OpenAI crawler documentation (developers.openai.com) |
User agent version numbers change over time. Match on the token, not the full string.
What ChatGPT-User does
ChatGPT-User is not an automatic crawler. It visits a page because a person asked ChatGPT something that needed it, or shared a link, or because an agent is completing a task. A visit from ChatGPT-User is often a sign that someone is reading about you inside ChatGPT right now.
Should you allow or block it?
Allow it. These visits come from real people asking about your topic.
If you block it: ChatGPT may be unable to read your page when a user asks about it, so answers can be thinner or rely on other sources.
- Training crawlers decide whether your content helps build future models. Blocking them does not remove you from AI search today.
- Search crawlers decide whether AI engines can find and cite your pages. Blocking them usually costs visibility.
- User triggered fetchers read a page because a person asked. Blocking them stops the AI reading you at the moment someone is interested.
How to find ChatGPT-User in your logs
Search your server or CDN access logs for the token ChatGPT-User. This command counts visits by IP, path and status code in a standard access log:
grep -i "ChatGPT-User" access.log | awk '{print $1, $7, $9}' | sort | uniq -c | sort -rn | head
A run of 403 or 503 status codes means your firewall or bot protection is blocking it, even if robots.txt allows it. That is one of the most common hidden reasons sites lose AI visibility.
How to tell if a visit is genuine
Anyone can put ChatGPT-User in a user agent string, so the name alone proves nothing. Check the request IP against openai.com/chatgpt-user.json.
Other OpenAI bots
- OAI-SearchBot: ai search crawler. Finds and indexes pages so they can appear and be cited in ChatGPT search.
- GPTBot: training crawler. Collects public web pages that may be used to train OpenAI's models.