Robots.txt generator
Configure search engine indexing and crawler permissions for Google, Bing, and AI scrapers. Add allow and disallow paths, configure XML sitemaps, and preview or download a compliant RFC 9309 robots.txt file. Everything runs in your browser without tracking.
Generates RFC 9309 robots.txt in your browser · No external requests
Crawler Rule Groups
Comma-separated or pick from presets
Sitemap & Host Directives
e.g. example.com (used by Yandex)
Generated robots.txt
# robots.txt generated locally by Plaintools (RFC 9309) # No server tracking or external requests User-agent: * Disallow: /admin/ Disallow: /private/ Disallow: /api/ Allow: / Sitemap: https://example.com/sitemap.xml
Where does robots.txt go?
Save this file as robots.txt and upload it to your website’s root directory (e.g. https://example.com/robots.txt).
Robots Exclusion Protocol (RFC 9309) rules are case-sensitive. The most specific matching path takes precedence in search engines like Google and Bing.
Example
Standard site blocking admin & internal API
User-agent: * with Disallow: /admin/, /private/, and /api/, Allow: /, and Sitemap: https://example.com/sitemap.xml. Plaintools generates the complete robots.txt file with instant copy and download.
Related: Schema markup generator, UTM generator, Open Graph generator, Hreflang generator
FAQ
- Does this robots.txt generator send my site structure to a server?
- No. The form, path definitions, and file generation operate 100% client-side in your browser. Nothing is ever sent to or processed by a server.
- Where should the robots.txt file be placed?
- The file must be placed in the top-level root directory of your website so that it is accessible at https://yourdomain.com/robots.txt. Subdirectory robots.txt files (e.g. /sub/robots.txt) are ignored by search engine crawlers.
- Can robots.txt be used to protect sensitive private information?
- No. A robots.txt file is publicly readable by anyone. It tells well-behaved search engines where not to crawl, but malicious actors and bad bots can read it to find hidden paths. Use proper server authentication and authorization for private content.
- How do I block AI scrapers like GPTBot or CCBot?
- Add a dedicated rule group specifying the crawler's user-agent name (e.g. User-agent: GPTBot, CCBot, anthropic-ai, Claude-Web, Google-Extended, Bytespider) followed by Disallow: /. Standard search engines will still index public pages.
- How do I build a compliant robots.txt without this page?
- Create a plain UTF-8 text file. For each bot or group, declare one or more 'User-agent: <name>' lines (use '*' for all bots). Add 'Disallow: <path>' for blocked prefixes and 'Allow: <path>' for permitted sub-paths. Paths must start with '/'. At the bottom, declare 'Sitemap: <absolute-https-url>'. Separate distinct user-agent blocks with a blank line. Deploy to the exact domain root as /robots.txt.