Robots.txt Tester
A robots.txt tester shows whether your robots.txt rules allow or block a URL for a given crawler. Paste your file, enter a user agent and up to 500 URLs or paths, and this free tool shows the result for each one and the exact rule that decided it. It follows the matching rules Google documents.
Test Results Summary
| Test URL / Path | Status | Deciding Rule | Rule Line # |
|---|
What is a robots.txt tester and when should you use it?
A tester answers one question for each URL: can this crawler fetch it under these rules? It reads your groups the same way Google describes, picks the group that matches the crawler, then compares every rule with the URL path and keeps the most specific match. Build or edit your rules with the Robots.txt Generator, and if you are moving pages, plan the moves with the Canonical Tag Generator.
How do you use it?
- Paste your robots.txt rules.
- Type the user agent, such as Googlebot.
- Paste the URLs or paths you want to check.
- Press Test.
- Read the winning rule for each URL.
Working on a comprehensive technical audit? Browse all free SEO tools to discover the next step in your workflow, or contact our technical support team if you require a specialized custom utility.
Result interpretation & practical applications
When evaluating the output of the robots.txt tester, review the parsed metrics against your specific site architecture. Outputs are formatted for direct implementation into your HTML source, web server configuration, or Google Search Console audit files.
Always inspect generated values for contextual nuance. Automated tools highlight structural patterns, but manual verification confirms that technical directives align with your overarching SEO strategy.
Technical rules and limitations
- Only one group applies to a crawler: the one with the most specific matching user agent; groups for the same user agent are merged.
- The most specific rule (the longest matching path) wins; when rules conflict, Google uses the least restrictive one.
- * matches any characters and $ marks the end of the URL; matching is case-sensitive.
- Rules without a path, and fields other than user-agent, allow, disallow and sitemap, are ignored.
These rules come from Google's robots.txt specification.
Common mistakes and troubleshooting
- Testing with the wrong user agent.
- Assuming the first rule in the file wins.
- Forgetting that /Fish and /fish are different paths.
- Expecting a disallow rule to remove a page from results.
No tool can promise how a search engine will crawl, index, or rank your pages. This tool generates and validates code against documented technical specifications; you determine where to deploy it, and you should always test changes on staging environments first. Google publishes extensive guidelines in Search Central on evaluating third-party SEO tools and advice. Learn more about our technical testing standards on our about us page, or explore full site audits on our pricing page.
Related free tools & workflows
Browse all Technical SEO Audit Tools →Technical SEO Audit Workflows
Indexability & Crawling
Redirects & HTTP Headers
Authoritative Technical Documentation & Standards
This tool executes logic aligned with official technical documentation and search engine webmaster standards:
- Google Search Central Robots.txt specification
Google Search Central: Robots.txt Specifications and Directives
Primary engineering source and technical documentation.
- IETF RFC 9309 Robots Exclusion Protocol
IETF RFC 9309: Robots Exclusion Protocol Standard
Primary engineering source and technical documentation.
Robots.txt Tester questions
Does the order of rules matter? ↓
No. The longest matching rule wins, and in a tie the allow rule wins.
Does it fetch my live robots.txt? ↓
No. Paste the file you want to test.
Why is my URL allowed when I disallowed its folder? ↓
A longer allow rule may match it. The table shows the winning rule.
Is this tool free? ↓
Yes. It is completely free to use and requires no registration or subscription.
Does this tool visit my website or store my files? ↓
No. It executes entirely in your local browser runtime. Your inputs and results are never stored or logged on our servers.
Ready for the next step in your audit?
ToxicBacklink – Free SEO Tools & Backlink Checker builds fast, browser-based utilities for webmasters, developers, and SEO professionals. Each tool executes locally, shows exact modifications, and follows published specifications. Read more about us, check our how it works guide, browse all tools, or return to the home page.