Free SEO Tools
Robots.txt Checker
Enter a domain to fetch and parse its robots.txt — every User-agent group, Allow/Disallow rule, and referenced sitemap.
What this checks
This tool fetches /robots.txt for the domain you enter and parses it into its User-agent groups, Allow/Disallow rules, and referenced sitemaps, flagging basic syntax problems along the way. It implements the common, de facto standard subset of the spec — not a full reproduction of any specific search engine's crawling behavior.
Frequently Asked Questions
- What does this tool check?
- Whether a given path is allowed or disallowed for a generic crawler, and lists any sitemap declarations found.
- Does robots.txt control indexing?
- No — robots.txt controls crawling, not indexing directly; a disallowed page can sometimes still be indexed from other signals.
- What if robots.txt is missing?
- A missing or unreachable robots.txt means crawling is generally unrestricted — it's not treated as a block.