We learned in Chapter 2, Customizing WordPress for SEO, that WordPress generates archive, tag, comment, and category pages that raise duplicate content issues. We can signal to search engines to ignore these duplicate content pages with a robots.txt file. In this section, we'll kill a few birds with one ultimate robots.txt file. We'll tell search engines to ignore our duplicated pages. We'll go further: we'll instruct search engines not to index our admin area and not to index non-essential folders on our server. As an option, we can also ask bad bots not to index any pages on our site, though they tend to usually do as they wish.

Before we customize our robots.txt file, bear in mind that the WordPress SEO by Yoast plugin can help us accomplish the same goal, but not as reliably. Furthermore, robots.txt is very handy for all sorts of purposes, so it's a great idea to know how to leverage it.

You can create a robots.txt file in any text editor. Place the...