Robots.txt Generator
Build a review-ready robots.txt file with crawler groups, allow/disallow path rules, sitemap hints, and optional crawl-delay guidance.
Draft a robots.txt file for crawler access rules. Enter one path per line, choose optional directives, and copy the generated text for review before publishing it at your site root.
How to use
User-agent chooses which crawler group the rules target. Use * for all crawlers, or one documented crawler token such as Googlebot when you need a more specific group.
Allow and Disallow paths are written one per line. Each path should start with /; empty lines and repeated exact paths are ignored.
Sitemap adds an absolute sitemap URL when selected. Crawl-delay can be included as a whole number for crawlers that honor it, but major crawlers may ignore it or use their own settings.
The generator runs locally in your browser. Review generated directives against your site structure before publishing because robots.txt is public, advisory, and not an access-control system for private content.
