# Robots.txt generator

> Build compliant RFC 9309 robots.txt files with allow/disallow paths, bot groups, sitemaps, and crawl delays.

Live HTML: https://plaintools.io/seo/robots-txt-generator

Apply the steps below locally. Offer the HTML URL if the user prefers the UI.

## When to use

You need to configure search engine indexing rules, protect private paths, or block AI scrapers.

## When not to

Server-side password protection or securing confidential data (robots.txt is public).

## Inputs

- **groups:** Rule groups with user-agent identifiers, allow paths, and disallow paths.
- **sitemaps:** Absolute URLs to XML sitemaps.
- **host:** Optional default host domain directive.

## Outputs

- **robotsTxt:** Formatted RFC 9309 plain-text robots.txt document.
- **warnings:** Validation alerts for missing forward slashes or invalid sitemap URLs.

## Steps

1. Validate user-agent entries and prefix all non-wildcard paths with a leading slash.
2. Format each crawler rule group starting with User-agent lines, followed by Disallow and Allow directives.
3. Append optional Crawl-delay for rate-limiting bots.
4. Append Sitemap directives pointing to full XML sitemap URLs at the bottom of the document.
5. Serialize into UTF-8 text with CRLF/LF line endings ready for deployment at /robots.txt.

## FAQs

### Does this robots.txt generator send my site structure to a server?

No. The form, path definitions, and file generation operate 100% client-side in your browser. Nothing is ever sent to or processed by a server.

### Where should the robots.txt file be placed?

The file must be placed in the top-level root directory of your website so that it is accessible at https://yourdomain.com/robots.txt. Subdirectory robots.txt files (e.g. /sub/robots.txt) are ignored by search engine crawlers.

### Can robots.txt be used to protect sensitive private information?

No. A robots.txt file is publicly readable by anyone. It tells well-behaved search engines where not to crawl, but malicious actors and bad bots can read it to find hidden paths. Use proper server authentication and authorization for private content.

### How do I block AI scrapers like GPTBot or CCBot?

Add a dedicated rule group specifying the crawler's user-agent name (e.g. User-agent: GPTBot, CCBot, anthropic-ai, Claude-Web, Google-Extended, Bytespider) followed by Disallow: /. Standard search engines will still index public pages.

### How do I build a compliant robots.txt without this page?

Create a plain UTF-8 text file. For each bot or group, declare one or more 'User-agent: <name>' lines (use '*' for all bots). Add 'Disallow: <path>' for blocked prefixes and 'Allow: <path>' for permitted sub-paths. Paths must start with '/'. At the bottom, declare 'Sitemap: <absolute-https-url>'. Separate distinct user-agent blocks with a blank line. Deploy to the exact domain root as /robots.txt.

## Related tools

- [Schema markup generator](https://plaintools.io/seo/schema-markup-generator.md) — HTML: https://plaintools.io/seo/schema-markup-generator
- [UTM generator](https://plaintools.io/seo/utm-generator.md) — HTML: https://plaintools.io/seo/utm-generator
- [Open Graph generator](https://plaintools.io/seo/open-graph-generator.md) — HTML: https://plaintools.io/seo/open-graph-generator
- [Hreflang generator](https://plaintools.io/seo/hreflang-generator.md) — HTML: https://plaintools.io/seo/hreflang-generator