Robots.txt Analyzer
Enter a site's URL to retrieve its robots.txt file and review the User-agent, Disallow, Allow and Sitemap directives it contains.
How to Use This Tool
Enter the site's homepage URL
Just the domain, like https://example.com — no need to add /robots.txt yourself.
Click "Check robots.txt"
The tool fetches example.com/robots.txt directly from the live site.
Review the directives
See a count of Disallow, Allow and Sitemap lines, plus the full raw file.
Fix accidental blocks
If something important is disallowed by mistake, update the file on your server.
About the Robots.txt Analyzer
robots.txt is a plain text file placed at the root of a website (like example.com/robots.txt) that tells well-behaved search engine crawlers which parts of the site they're allowed or disallowed from crawling. It's one of the first files most crawlers check before indexing a site.
A misconfigured robots.txt is one of the most damaging technical SEO mistakes possible, because a single accidental "Disallow: /" can block an entire site from being crawled — sometimes without anyone noticing for weeks, since the site still loads fine for human visitors.
This tool fetches the live robots.txt file from any domain you enter, counts how many Disallow, Allow and Sitemap directives it contains, and shows you the full raw file content so you can quickly verify nothing important is being blocked.
Frequently Asked Questions
Is robots.txt required for every website?
No, it's optional — if a site has no robots.txt file, crawlers generally assume they're allowed to crawl everything. It becomes important specifically when you want to restrict access to certain sections, like admin areas or duplicate content.
Can robots.txt stop a page from appearing in Google search results?
Not reliably on its own. Disallowing a page in robots.txt only blocks crawling, but Google can still index a URL it discovers through external links without ever fetching its content — showing it in results with no description. To truly prevent indexing, use a noindex meta tag instead.
Why might this tool fail to retrieve a site's robots.txt?
Some servers block cross-origin requests from browser-based tools like this one (a security setting called CORS), which prevents the fetch from completing even though the file exists and loads fine when you visit it directly in a new tab.
What does "Disallow: /" mean, and why is it dangerous?
It tells crawlers not to access anything on the entire site starting from the root. It's a common accidental mistake — often left over from a staging environment — that can completely block a live site from being crawled or indexed.
Should my sitemap URL be listed in robots.txt?
It's a widely recommended best practice. Adding a "Sitemap:" line pointing to your XML sitemap in robots.txt helps search engines discover it quickly, even in addition to submitting it manually through tools like Google Search Console.
Can different crawlers (like Googlebot vs Bingbot) be given different rules?
Yes — robots.txt supports separate User-agent blocks, so you can write different Allow/Disallow rules for Googlebot specifically versus other crawlers, or use "User-agent: *" to apply a rule to all crawlers at once.