Robots.txt Generator Pro
Take control of how search engines crawl and index your site with perfectly crafted directives.
Some tools can take a bit log time to load properly on the background, please wait until it's complete.
General Settings
User-Agent Specific Directives
*
Add New User-Agent Block
Generated robots.txt
Understanding robots.txt
What is robots.txt?
A `robots.txt` file tells search engine crawlers which pages or files the crawler can or can't request from your site. It's placed in the root directory of your website (e.g., `https://example.com/robots.txt`).
Key Directives
- User-agent: Specifies the web crawler (e.g., `Googlebot`, `*` for all).
- Disallow: Tells a user-agent not to crawl particular URLs or directories. An empty `Disallow:` means allow all.
- Allow: Explicitly permits crawling of a URL or directory, even if its parent directory is disallowed.
- Sitemap: Provides the location of your XML sitemap(s).
- Crawl-delay: (Less common now, some bots ignore it) Specifies how many seconds a crawler should wait between requests.
Important: `robots.txt` is a guideline, not an enforcement mechanism. Malicious bots may ignore it. To truly protect private content, use server-side authentication.