AI Crawler Allow Status Checker (Search vs. Training Bots)
Paste a robots.txt to see which of 20 search, answer and training bots may read the site.
Separates bots that bring citations from training-only crawlers, with the rule that decides each one.
How to use this tool
Shows which of 20 search, user-fetch and training bots can currently read a site, based on the robots.txt you paste. Use it to find out why a bot may not be reaching your pages.
- Paste your live robots.txt into the Your robots.txt box.
- Optionally paste your page meta tags into the meta tags box.
- Check each bot card for ALLOWED or BLOCKED and the rule that decided it.
- Test a fix with the Allow and Disallow buttons, then press Copy robots.txt.
What it checks
- Allowed or blocked status for each of the 20 bots under your robots.txt.
- The deciding robots.txt line, or a note that no rule applies.
- Meta robots directives that block a bot even when robots.txt allows it.
- A count of blocked search and answer bots and of blocked training bots.
- A verdict naming the search or answer bots that cannot read the site.
Questions
Does it fetch my robots.txt for me?
No. You paste the file. The URL scan box does not load it into this tool.
Why is a bot allowed with no rule?
Under RFC 9309, a crawler may fetch a path when no rule matches it.
Why is Grok not listed?
xAI publishes no robots.txt token, so the tool leaves it out.