Generate a robots.txt file to guide search engines on crawling your site.
A robots.txt file tells search engine crawlers which URLs the crawler can access on your site. This is used mainly to avoid overloading your site with requests or to keep certain private directories out of the public search results.
Our generator helps you write syntactically correct rules without needing to memorize the directives. Simply input your restricted directories and sitemap URL to generate a file you can place at the root of your domain.