Robots.txt Generator
Use * for all bots, or a specific name like Googlebot
Save this content in a file named robots.txt at the root of your site.
Frequently Asked Questions
Everything you need to know about the robots.txt file
What is the robots.txt file for?
The robots.txt file tells search engine crawlers which parts of the site they can or cannot crawl. It must be placed at the root of the domain (e.g. https://example.com/robots.txt). Note: it is a directive, not a security mechanism; a blocked page can still be indexed if other sites link to it.
What is the difference between Allow and Disallow?
Disallow blocks crawler access to a path, while Allow explicitly permits it. Allow is useful for creating exceptions: for example you can block /admin/ but allow /admin/public/. An empty "Disallow:" line allows access to the whole site, while "Disallow: /" blocks the entire site.
Do all search engines respect crawl-delay?
No. The Crawl-delay directive tells bots how many seconds to wait between requests, but not all of them respect it. Bing and Yandex support it, while Google ignores it: for Google the crawl rate is managed through Google Search Console. Only use crawl-delay if your server has load problems.
What is a robots.txt file for?
It tells the crawlers of the search engines which parts of the site they may walk through. It goes in the root of the domain, and it protects nothing: a blocked page can still be reached by anyone who knows its address.
Do all the search engines honour crawl delay?
No. Bing and Yandex take it into account, Google ignores it and the crawl rate is set from Search Console instead. It stays useful for holding back the more aggressive crawlers.