On the Internet, websites and their business models mostly rely on search engines. Say, for example, when a user is searching for the keyword "book publishing" in the Google search engine, if Packt's website is listed on the first page of results, it will be an advantage for Packt—especially when its business model relies on Internet users.

Search engines, such as Google, order the result page based on a number of factors, referred to as algorithms. These are: keyword density on the page, trusted inwards links to the page, popularity of the website, and so on. All these depend on how far the search engine's spider can crawl (or reach) the website's content. If the website's index page doesn't have links to the website's inner pages, has restricted access to inner pages, or doesn't expose the inner pages through the sitemap.xml file that the spider looks for when crawling, the contents will not be indexed and can't be searched.

The challenge with Web 2.0...