Does this robots.txt generator send my site structure to a server?

No. The form, path definitions, and file generation operate 100% client-side in your browser. Nothing is ever sent to or processed by a server.

Where should the robots.txt file be placed?

The file must be placed in the top-level root directory of your website so that it is accessible at https://yourdomain.com/robots.txt. Subdirectory robots.txt files (e.g. /sub/robots.txt) are ignored by search engine crawlers.

Can robots.txt be used to protect sensitive private information?

No. A robots.txt file is publicly readable by anyone. It tells well-behaved search engines where not to crawl, but malicious actors and bad bots can read it to find hidden paths. Use proper server authentication and authorization for private content.

How do I block AI scrapers like GPTBot or CCBot?

Add a dedicated rule group specifying the crawler's user-agent name (e.g. User-agent: GPTBot, CCBot, anthropic-ai, Claude-Web, Google-Extended, Bytespider) followed by Disallow: /. Standard search engines will still index public pages.

How do I build a compliant robots.txt without this page?

Create a plain UTF-8 text file. For each bot or group, declare one or more 'User-agent: ' lines (use '*' for all bots). Add 'Disallow: ' for blocked prefixes and 'Allow: ' for permitted sub-paths. Paths must start with '/'. At the bottom, declare 'Sitemap: '. Separate distinct user-agent blocks with a blank line. Deploy to the exact domain root as /robots.txt.

Does Disallow in robots.txt remove a webpage from Google search results?

No. A Disallow directive tells search engine crawlers not to fetch the page, but Google can still index the URL if other websites link to it. To guarantee a page is removed from search results, allow it to be crawled and include a 'noindex' meta tag in its HTML.

Robots.txt generator

Configure search engine indexing and crawler permissions for Google, Bing, and AI scrapers. Add allow and disallow paths, configure XML sitemaps, and preview or download a compliant RFC 9309 robots.txt file. Everything runs in your browser without tracking.

Generates RFC 9309 robots.txt in your browser · No external requests

Presets:

Crawler Rule Groups

Group #1

User-Agent

Comma-separated or pick from presets

Disallow Paths (Blocked)

Disallow: /admin/Disallow: /private/Disallow: /api/

Allow Paths (Permitted)

Allow: /

Crawl-Delay (Seconds, optional)

Sitemap & Host Directives

XML Sitemap URL(s)

https://example.com/sitemap.xml

Host Directive (Optional)

e.g. example.com (used by Yandex)

Generated robots.txt

# robots.txt generated locally by Plaintools (RFC 9309)
# No server tracking or external requests

User-agent: *
Disallow: /admin/
Disallow: /private/
Disallow: /api/
Allow: /

Sitemap: https://example.com/sitemap.xml

Where does robots.txt go?

Save this file as robots.txt and upload it to your website’s root directory (e.g. https://example.com/robots.txt).

Robots Exclusion Protocol (RFC 9309) rules are case-sensitive. The most specific matching path takes precedence in search engines like Google and Bing.