Robots.txt Generator Pro

Take control of how search engines crawl and index your site with perfectly crafted directives.

Some tools can take a bit log time to load properly on the background, please wait until it's complete.

General Settings

User-Agent Specific Directives

*

Add New User-Agent Block

Generated robots.txt

Understanding robots.txt

What is robots.txt?

A `robots.txt` file tells search engine crawlers which pages or files the crawler can or can't request from your site. It's placed in the root directory of your website (e.g., `https://example.com/robots.txt`).

Key Directives

  • User-agent: Specifies the web crawler (e.g., `Googlebot`, `*` for all).
  • Disallow: Tells a user-agent not to crawl particular URLs or directories. An empty `Disallow:` means allow all.
  • Allow: Explicitly permits crawling of a URL or directory, even if its parent directory is disallowed.
  • Sitemap: Provides the location of your XML sitemap(s).
  • Crawl-delay: (Less common now, some bots ignore it) Specifies how many seconds a crawler should wait between requests.

Important: `robots.txt` is a guideline, not an enforcement mechanism. Malicious bots may ignore it. To truly protect private content, use server-side authentication.