We learned in Chapter 2, Customizing WordPress Settings for SEO, that WordPress generates archive, tag, comment, and category pages that raise duplicate content issues. We can signal to search engines to ignore these duplicate content pages with a robots.txt file. In this section, we'll kill a few birds with one ultimate robots.txt file. We'll tell search engines to ignore our duplicated pages. We'll go further: we'll instruct search engines not to index our admin area and not to index non-essential folders on our server. As an option, we can also ask bad bots not to index any pages on our site, though they tend to usually do as they wish.
You can create a robots.txt file in any text editor. Place the file in the root directory/folder of your website (not the WordPress template folder) and the search engines will find it automatically.
The following robots.txt is quite simple, but can accomplish much in a few lines:
User-agent: * Disallow: /cgi-bin Disallow...