Robots.txt Generator
Create a valid robots.txt file to guide search engine crawlers. Specify User-agent rules, Disallow admin paths, and link your XML sitemap URL.
Step-by-Step Instructions
Select Search Engine User-Agents
Choose default `*` (all robots) or specify rules for Googlebot, Bingbot, Slurp, DuckDuckBot, or AI crawlers (GPTBot, CCBot).
Define Allow & Disallow Rules
Specify directory paths to block (e.g., `/admin/`, `/cart/`, `/api/`) and paths to allow.
Include XML Sitemap URL
Input the full absolute URL to your XML sitemap (`https://example.com/sitemap.xml`).
Copy or Download robots.txt
Download the formatted `robots.txt` file and upload it to the root directory of your web server.
Key Features & Capabilities
- Robots Exclusion Protocol (RFC 9309) compliant robots.txt generator
- Targeted AI crawler directives to block or allow LLM scrapers (GPTBot, ClaudeBot, CCBot, Google-Extended, Bytespider)
- Crawl-Delay parameter configuration for rate-limiting aggressive crawlers
- Sitemap index URL integration and wildcards pattern matching (`*` and `$`) support
Frequently Asked Questions
What is the purpose of a robots.txt file?
A `robots.txt` file is a plain text file placed in the root directory of your website (`https://example.com/robots.txt`) that instructs search engine web crawlers and automated bots which pages or sections of your site they are allowed or forbidden to crawl under the Robots Exclusion Protocol (RFC 9309).
Does Disallow in robots.txt prevent a page from being indexed by Google?
No! A common SEO misconception: `Disallow: /private/` prevents Google from *crawling* the page, but if other external websites link to that URL, Google can still index the URL in search results without reading its content. To guarantee a page is never indexed, allow crawling and place a `<meta name="robots" content="noindex">` tag on the page or return an `X-Robots-Tag: noindex` HTTP header.
How do you block AI scrapers (like OpenAI and Anthropic) in robots.txt?
You can explicitly block AI training crawlers by adding targeted user-agent directives: `User-agent: GPTBot`, `User-agent: ClaudeBot`, `User-agent: CCBot`, `User-agent: Google-Extended`, followed by `Disallow: /`.
Where must the robots.txt file be uploaded on a web server?
The `robots.txt` file must be located in the top-level root directory of the domain (e.g., `https://example.com/robots.txt`). Placing it in a subdirectory (like `example.com/assets/robots.txt`) will cause search engines to ignore it completely.
Related Utilities
Discover complementary online tools in the same workflow.
JSON Formatter & Validator
Format, beautify, minify, validate, and view JSON data online with instant syntax highlighting and error detection.
Base64 Encoder & Decoder
Encode text or binary data into Base64 format or decode Base64 back into raw string format.
CSV to JSON Converter
Easily convert CSV data to clean JSON format. Turn tabular spreadsheets into structured JSON.
YAML to JSON Converter
Convert YAML configuration files into structured JSON for APIs and modern web tooling.
JWT Parser & Token Debugger
Decode JWT tokens to view header, payload claims, and expiration status without sending secrets anywhere.
Hash Generator (SHA-256, SHA-512, MD5)
Generate cryptographically secure SHA-256, SHA-512, SHA-384, and SHA-1 hashes for any text.