One last technique that can benefit your scraper, as well as reducing load on the website, is by only requesting new content when it changes. If your scraper is downloading the same old content from a web server, then you aren't getting any new information and the web server is doing unnecessary work. For this reason, most web servers implement techniques to provide the client with instructions on caching.
A website that supports caching, will give the client information on what it can store, and how long to store it. This is done through response headers such as Cache-Control, Etag, Date, Expires, and Vary. Your web scraper should be aware of these directives to avoid making unnecessary requests to the web server, saving you, and the server, time and computing resources. Let's take a look at our http://www.example.com/index.html response one more...