AI Crawler Rules Generator
Select known AI crawlers to generate robots.txt rules blocking or allowing them — entirely in your browser.
GPTBot
OpenAI — Used to train OpenAI's models.
ChatGPT-User
OpenAI — Fetches pages a user references in ChatGPT.
OAI-SearchBot
OpenAI — Powers ChatGPT search results.
ClaudeBot
Anthropic — Used to train Anthropic's Claude models.
Claude-Web
Anthropic — Fetches pages referenced in Claude.
Google-Extended
Google — Controls use for Gemini and Vertex AI training, separate from standard Googlebot.
PerplexityBot
Perplexity — Crawls for Perplexity's AI search answers.
Applebot-Extended
Apple — Controls use for Apple's generative AI training.
Bytespider
ByteDance — Used to train ByteDance's AI models.
CCBot
Common Crawl — A general web archive widely used as AI training data.
Meta-ExternalAgent
Meta — Used to train Meta's AI models.
cohere-ai
Cohere — Used by Cohere's AI systems.
About this tool
These rules are entirely under your control and only take effect once you add them to your own robots.txt and publish it. robots.txt is a voluntary convention — it only affects crawlers that choose to honor it. Verify your published file with the AI Crawler Access Checker.
Frequently Asked Questions
- What does this generate?
- robots.txt snippets to block or explicitly allow selected AI crawlers.
- Do these rules take effect immediately?
- No, they're entirely under your control and only take effect once you add them to your own robots.txt and publish it.
- Can I verify my published rules?
- Yes, use the AI Crawler Access Checker afterward.