robots.txt rules
Copy the group you need into the robots.txt file at the root of your site. Keep only one option per bot.
At a glance
| Detail | Bytespider |
|---|---|
| Operator | ByteDance |
| Type | Training crawler |
| What it does | A ByteDance crawler widely linked to data collection for its AI products. |
| robots.txt token | Bytespider |
| Example user agent | Contains "Bytespider" |
| Follows robots.txt | There is no official statement. Site owners have reported it fetching pages that robots.txt disallows. |
| How to verify | There is no published IP list. Rely on logs and firewall rules. |
| If you block it | If you block it at the firewall, ByteDance cannot collect your content through this crawler. |
| Source | No official documentation published |
User agent version numbers change over time. Match on the token, not the full string.
What Bytespider does
Bytespider is a crawler operated by ByteDance, the company behind TikTok. ByteDance does not publish public documentation for it, so what is known comes from site owners' logs. It is among the AI crawlers most often blocked in robots.txt files.
Should you allow or block it?
Many sites block it, often at the firewall as well as in robots.txt, because its behaviour is not documented.
If you block it: If you block it at the firewall, ByteDance cannot collect your content through this crawler.
- Training crawlers decide whether your content helps build future models. Blocking them does not remove you from AI search today.
- Search crawlers decide whether AI engines can find and cite your pages. Blocking them usually costs visibility.
- User triggered fetchers read a page because a person asked. Blocking them stops the AI reading you at the moment someone is interested.
How to find Bytespider in your logs
Search your server or CDN access logs for the token Bytespider. This command counts visits by IP, path and status code in a standard access log:
grep -i "Bytespider" access.log | awk '{print $1, $7, $9}' | sort | uniq -c | sort -rn | head
A run of 403 or 503 status codes means your firewall or bot protection is blocking it, even if robots.txt allows it. That is one of the most common hidden reasons sites lose AI visibility.
How to tell if a visit is genuine
Anyone can put Bytespider in a user agent string, so the name alone proves nothing. There is no published IP list. Rely on logs and firewall rules.