Robots.txt Generator
A robots.txt generator builds the robots.txt file that tells crawlers which parts of your site they may crawl. Add user-agent groups, allow and disallow paths and your sitemap URLs, and this free tool writes a correctly formatted file you can copy or download. It also warns you about rules that Google ignores.
User-Agent Groups
Generated robots.txt
0 bytes / 512,000 maxWhat is a robots.txt generator and when should you use it?
A robots.txt file is a plain text file at the top level of your site that lists rules for crawlers. Each group starts with a user-agent line, followed by allow and disallow rules for paths on your site. It controls crawling, not indexing, so it is not the right tool for keeping a page out of search results. To check how your rules treat a specific URL, use the Robots.txt Tester, and to list your pages for search engines, use the XML Sitemap Generator.
How do you use it?
- Add a group and type a user agent, such as * for all crawlers.
- Add allow and disallow paths, or use a preset.
- Add your sitemap URLs.
- Copy or download the file.
- Upload it as robots.txt in the top-level folder of your site.
Working on a comprehensive technical audit? Browse all free SEO tools to discover the next step in your workflow, or contact our technical support team if you require a specialized custom utility.
Result interpretation & practical applications
When evaluating the output of the robots.txt generator, review the parsed metrics against your specific site architecture. Outputs are formatted for direct implementation into your HTML source, web server configuration, or Google Search Console audit files.
Always inspect generated values for contextual nuance. Automated tools highlight structural patterns, but manual verification confirms that technical directives align with your overarching SEO strategy.
Technical rules and limitations
- The file must be in the top-level directory of the site and applies only to that host, protocol and port.
- It must be UTF-8 plain text; Google ignores content after 500 KiB.
- Google supports four fields: user-agent, allow, disallow and sitemap; other fields such as crawl-delay are not supported.
- Paths start with /, are case-sensitive and can use * (any characters) and $ (end of the URL).
- A disallowed page can still appear in results as a URL without a snippet; use noindex to keep a page out of results.
These rules come from Google's robots.txt specification.
Common mistakes and troubleshooting
- Blocking the whole site with Disallow: / by accident.
- Putting the file in a subfolder, where crawlers do not look.
- Using robots.txt to hide a page from search results.
- Writing paths without a leading /.
- Relying on crawl-delay for Google.
No tool can promise how a search engine will crawl, index, or rank your pages. This tool generates and validates code against documented technical specifications; you determine where to deploy it, and you should always test changes on staging environments first. Google publishes extensive guidelines in Search Central on evaluating third-party SEO tools and advice. Learn more about our technical testing standards on our about us page, or explore full site audits on our pricing page.
Related free tools & workflows
Browse all Technical SEO Audit Tools →Technical SEO Audit Workflows
Indexability & Crawling
Redirects & HTTP Headers
Authoritative Technical Documentation & Standards
This tool executes logic aligned with official technical documentation and search engine webmaster standards:
- Google Search Central Robots.txt specification
Google Search Central: Robots.txt Specifications and Directives
Primary engineering source and technical documentation.
- IETF RFC 9309 Robots Exclusion Protocol
IETF RFC 9309: Robots Exclusion Protocol Standard
Primary engineering source and technical documentation.
Robots.txt Generator questions
Where do I put robots.txt? ↓
In the top-level folder of your site, so it opens at /robots.txt. A file on a subdomain only applies to that subdomain.
Does disallow remove a page from Google? ↓
No. It stops crawling. A blocked URL can still appear without a snippet; use noindex instead.
Can I add my sitemap? ↓
Yes. Add one or more Sitemap lines with full URLs.
Is this tool free? ↓
Yes. It is completely free to use and requires no registration or subscription.
Does this tool visit my website or store my files? ↓
No. It executes entirely in your local browser runtime. Your inputs and results are never stored or logged on our servers.
Ready for the next step in your audit?
ToxicBacklink – Free SEO Tools & Backlink Checker builds fast, browser-based utilities for webmasters, developers, and SEO professionals. Each tool executes locally, shows exact modifications, and follows published specifications. Read more about us, check our how it works guide, browse all tools, or return to the home page.