As an HTTP client developer, you may have different priorities to the webmasters that run websites. A webmaster will typically provide a site for human users; possibly offering a service designed for generating revenue, and it is most likely that all this will need to be done with the help of very limited resources. They will be interested in analyzing how humans use their site, and may have areas of the site they would prefer that automated clients didn't explore.

HTTP clients that automatically parse and download pages on websites are called various things, such as bots, web crawlers, and spiders. Bots have many legitimate uses. All the search engine providers make extensive use of bots for crawling the web and building their huge page indexes. Bots can be used to check for dead links, and to archive sites for repositories, such as the Wayback Machine. But, there are also many uses that might be considered as illegitimate. Automatically traversing an information service...