AI Crawler vs. AI Search Policy Matrix & Robots.txt Builder
Compare AI Training Crawlers (GPTBot, CCBot, Google-Extended) vs. Live AI Search Citation Bots (OAI-SearchBot, PerplexityBot, Claude-SearchBot).
Checks 20 crawler and AI agent tokens as their operators document them, grouped into search, user fetch and training, and builds the robots.txt rules.
How to use this tool
Helps you pick a robots.txt policy for AI bots: let search and answer bots in, keep training bots out, or allow everything. It covers 20 bots in three groups.
- Press Allow search and answers, block training, or Allow every bot, to load a starting robots.txt.
- Use the Allow and Disallow buttons on a bot card to change one bot.
- Read the verdict and the two counters for blocked search bots and blocked training bots.
- Press Copy robots.txt and publish it at your site root.
What it checks
- Bots grouped as search index, opens pages for a user, and model training.
- Whether each bot may fetch the site root under your rules, using RFC 9309 matching.
- The robots.txt line that decided each result.
- What you lose by blocking each bot.
- Optional meta tags with noindex, none or nosnippet, and a warning that noai is not a standard.
Questions
What is the difference between search and training bots?
Search bots fetch pages so an AI product can show or cite them. Training bots collect content to build models. You can allow the first group and block the second.
Does blocking a training bot hide me from AI search?
The tool lists search and training tokens separately, so blocking a training token does not block the search token of the same company.
Can robots.txt force a bot to obey?
No. It is a request that well-behaved crawlers follow.