Robots.txt Generator

Disallow: /path — blocks crawling of that path and everything under it.

Allow: /path — carves out an exception inside a disallowed path.

Disallow: (empty) — allows everything for that user-agent.

Sitemap: — points crawlers to your XML sitemap; can appear once, anywhere in the file.

Remember: robots.txt only tells well-behaved crawlers not to fetch a path — it does not prevent indexing on its own, and it isn't a security control. Use meta robots noindex or password protection for anything that truly must stay out of search results.