robots.txt checker for AI

Check any site's robots.txt, or paste your own. We flag mistakes, show which AI crawlers are blocked, and write a corrected file.

In the suggested file, block bots that only collect training data (search stays open)

How do AI crawlers read robots.txt?

Each crawler uses the group that names it; if none does, it uses the * group. Within a group the longest matching rule wins, and Allow wins a tie.

No. Allowing AI search crawlers does not affect Google rankings. Blocking Google-Extended or GPTBot only stops model training, not search.

At the root of your domain, for example example.com/robots.txt. Each subdomain needs its own.

It is a request, not a lock. Check your firewall too with the AI crawler checker.

AI agent readiness checker

Can ChatGPT agent, Google AI Mode and Perplexity find, understand and buy from your site?

AI crawler access checker

See which AI crawlers your robots.txt and firewall let in, page by page.

llms.txt generator

Build an llms.txt file for your site from its sitemap in seconds.

Schema generator

Write Organization, Product, FAQ, Article and Local Business JSON-LD.

AI visibility check

Does ChatGPT recommend you or your competitor? Find out in 30 seconds.