Robots.txt Generator

What is robots.txt?

robots.txt is a small text file at the root of your site (example.com/robots.txt) that tells search-engine crawlers which parts of your site they may or may not access. This tool builds a valid robots.txt from a few choices — allow everything, block everything, or set custom Disallow rules — plus your sitemap location.

The basics

• User-agent — which crawler the rules apply to. * means all of them.
• Disallow — a path crawlers should not fetch (e.g. /admin/). An empty Disallow: means "allow everything."
• Sitemap — pointing crawlers to your sitemap helps them discover your pages.

FAQ

Does Disallow keep a page out of Google?

Not reliably. Disallow stops crawling, but a blocked URL can still be indexed if other sites link to it. To keep a page out of search results, use a noindex meta tag instead — and don't block it in robots.txt, or Google can't see the noindex.

Is this robots.txt generator free to use?

Yes, it's completely free with no sign-up, and you can generate as many robots.txt files as you need.

Where do I put the generated robots.txt file?

It must live at the root of your domain, at example.com/robots.txt. Crawlers only check that exact location, so a robots.txt placed in a subfolder will be ignored.

Will this block bad bots or scrapers?

No. robots.txt is a voluntary instruction that well-behaved crawlers like Googlebot respect, but malicious bots and scrapers can simply ignore it. To truly block traffic you need server-side rules such as firewall or authentication settings.

Should I add my sitemap to robots.txt?

Yes, it's good practice. Adding a Sitemap: line pointing to your sitemap URL helps crawlers discover all your pages, and this tool can include it for you when you provide the address.

Is anything I enter sent to a server?

No. The robots.txt is built entirely in your browser from your choices, so the paths and sitemap URL you type are never uploaded anywhere.