Free SEO Tools
AI Bot Robots.txt Analyzer
Enter a domain for a full table of every known AI bot, its exact matched robots.txt rule, and whether that rule is bot-specific or a generic fallback.
About this tool
This is the detailed diagnostic view — for a quick pass/fail summary, use the AI Crawler Access Checker. “Rule source” shows whether a bot has its own dedicated robots.txt block or is only covered by the generic * rule.
Frequently Asked Questions
- What does 'rule source' mean?
- Whether a bot has its own dedicated robots.txt block or is only covered by the generic * rule.
- How is this different from AI Crawler Access Checker?
- That tool gives a quick pass/fail summary; this one is the detailed diagnostic table with the exact matched rule per bot.
- Does this affect crawlers that don't honor robots.txt?
- No, robots.txt is a voluntary convention — it only affects crawlers that choose to honor it.