Free SEO Tools
Robots.txt Checker
Enter a domain to fetch and parse its robots.txt — every User-agent group, Allow/Disallow rule, and referenced sitemap.
What this checks
This tool fetches /robots.txt for the domain you enter and parses it into its User-agent groups, Allow/Disallow rules, and referenced sitemaps, flagging basic syntax problems along the way. It implements the common, de facto standard subset of the spec — not a full reproduction of any specific search engine's crawling behavior.