Book Image

Web Scraping with Python

By : Richard Penman
Book Image

Web Scraping with Python

By: Richard Penman

Overview of this book

Table of Contents (16 chapters)

Sequential crawler

Here is the code to use AlexaCallback with the link crawler developed earlier to download sequentially:

scrape_callback = AlexaCallback()

This code is available at and can be run from the command line as follows:

$ time python

This time is as expected for sequential downloading with an average of ~1.6 seconds per URL.