robots.txt rules
Copy the group you need into the robots.txt file at the root of your site. Keep only one option per bot.
At a glance
| Detail | Bingbot |
|---|---|
| Operator | Microsoft |
| Type | AI search crawler |
| What it does | Crawls pages for Bing, which grounds many Microsoft Copilot answers. |
| robots.txt token | Bingbot |
| Example user agent | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36 |
| Follows robots.txt | Microsoft says Bingbot follows robots.txt. |
| How to verify | Use the Verify Bingbot tool in Bing Webmaster Tools or a reverse DNS lookup to search.msn.com. |
| If you block it | Your site can drop out of Bing and reduce your presence in Copilot answers. |
| Source | Bing Webmaster Tools help |
User agent version numbers change over time. Match on the token, not the full string.
What Bingbot does
Bingbot is Microsoft's search crawler. Copilot grounds many web answers in Bing results, so being well indexed in Bing helps you appear in Copilot. Other services that use Bing's index benefit too.
Should you allow or block it?
Allow it, and submit your sitemap in Bing Webmaster Tools.
If you block it: Your site can drop out of Bing and reduce your presence in Copilot answers.
- Training crawlers decide whether your content helps build future models. Blocking them does not remove you from AI search today.
- Search crawlers decide whether AI engines can find and cite your pages. Blocking them usually costs visibility.
- User triggered fetchers read a page because a person asked. Blocking them stops the AI reading you at the moment someone is interested.
How to find Bingbot in your logs
Search your server or CDN access logs for the token Bingbot. This command counts visits by IP, path and status code in a standard access log:
grep -i "Bingbot" access.log | awk '{print $1, $7, $9}' | sort | uniq -c | sort -rn | head
A run of 403 or 503 status codes means your firewall or bot protection is blocking it, even if robots.txt allows it. That is one of the most common hidden reasons sites lose AI visibility.
How to tell if a visit is genuine
Anyone can put Bingbot in a user agent string, so the name alone proves nothing. Use the Verify Bingbot tool in Bing Webmaster Tools or a reverse DNS lookup to search.msn.com.