The Internet contains the most useful set of data ever assembled, which is largely publicly accessible for free. However, this data is not easily reusable. It is embedded within the structure and style of websites and needs to be extracted to be useful. This process of extracting data from web pages is known as web scraping and is becoming increasingly useful as ever more information is available online.
Chapter 1, Introduction to Web Scraping, introduces web scraping and explains ways to crawl a website.
Chapter 2, Scraping the Data, shows you how to extract data from web pages.
Chapter 3, Caching Downloads, teaches you how to avoid redownloading by caching results.
Chapter 4, Concurrent Downloading, helps you to scrape data faster by downloading in parallel.
Chapter 5, Dynamic Content, shows you how to extract data from dynamic websites.
Chapter 6, Interacting with Forms, shows you how to work with forms to access the data you are after.
Chapter 7, Solving CAPTCHA, elaborates how to access data that is protected by CAPTCHA images.
Chapter 8, Scrapy, teaches you how to use the popular high-level Scrapy framework.
Chapter 9, Overview, is an overview of web scraping techniques that have been covered.
All the code used in this book has been tested with Python 2.7, and is available for download at http://bitbucket.org/wswp/code. Ideally, in a future version of this book, the examples will be ported to Python 3. However, for now, many of the libraries required (such as Scrapy/Twisted, Mechanize, and Ghost) are only available for Python 2. To help illustrate the crawling examples, we created a sample website at http://example.webscraping.com. This website limits how fast you can download content, so if you prefer to host this yourself the source code and installation instructions are available at http://bitbucket.org/wswp/places.
We decided to build a custom website for many of the examples used in this book instead of scraping live websites, so that we have full control over the environment. This provides us stability—live websites are updated more often than books, and by the time you try a scraping example, it may no longer work. Also, a custom website allows us to craft examples that illustrate specific skills and avoid distractions. Finally, a live website might not appreciate us using them to learn about web scraping and try to block our scrapers. Using our own custom website avoids these risks; however, the skills learnt in these examples can certainly still be applied to live websites.
This book requires prior programming experience and would not be suitable for absolute beginners. When practical we will implement our own version of web scraping techniques so that you understand how they work before introducing the popular existing module. These examples will assume competence with Python and installing modules with pip. If you need a brush up, there is an excellent free online book by Mark Pilgrim available at http://www.diveintopython.net. This is the resource I originally used to learn Python.
The examples also assume knowledge of how web pages are constructed with HTML and updated with JavaScript. Prior knowledge of HTTP, CSS, AJAX, WebKit, and MongoDB would also be useful, but not required, and will be introduced as and when each technology is needed. Detailed references for many of these topics are available at http://www.w3schools.com.
In this book, you will find a number of styles of text that distinguish between different kinds of information. Here are some examples of these styles, and an explanation of their meaning.
Code words in text are shown as follows: "Most websites define a robots.txt file to let robots know any restrictions about crawling their website."
A block of code is set as follows:
<?xml version="1.0" encoding="UTF-8"?> <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> <url><loc>http://example.webscraping.com/view/Afghanistan-1</loc></url> <url><loc>http://example.webscraping.com/view/Aland-Islands-2</loc></url> <url><loc>http://example.webscraping.com/view/Albania-3</loc></url> ... </urlset>
When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold:
def link_crawler(..., scrape_callback=None):
…
links = []
if scrape_callback:
links.extend(scrape_callback(url, html) or [])
...Any command-line input or output is written as follows:
$ python performance.py Regular expressions: 5.50 seconds BeautifulSoup: 42.84 seconds Lxml: 7.06 seconds
New terms and important words are shown in bold. Words that you see on the screen, in menus or dialog boxes for example, appear in the text like this: " When regular users open this web page in their browser, they will enter their e-mail and password, and click on the Log In button to submit the details to the server."
Feedback from our readers is always welcome. Let us know what you think about this book—what you liked or may have disliked. Reader feedback is important for us to develop titles that you really get the most out of.
To send us general feedback, simply send an e-mail to <[email protected]>, and mention the book title through the subject of your message.
If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, see our author guide on www.packtpub.com/authors.
Now that you are the proud owner of a Packt book, we have a number of things to help you to get the most from your purchase.
Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you find a mistake in one of our books—maybe a mistake in the text or the code—we would be grateful if you would report this to us. By doing so, you can save other readers from frustration and help us improve subsequent versions of this book. If you find any errata, please report them by visiting http://www.packtpub.com/support, selecting your book, clicking on the errata submission form link, and entering the details of your errata. Once your errata are verified, your submission will be accepted and the errata will be uploaded to our website, or added to any list of existing errata, under the Errata section of that title.
Piracy of copyright material on the Internet is an ongoing problem across all media. At Packt, we take the protection of our copyright and licenses very seriously. If you come across any illegal copies of our works, in any form, on the Internet, please provide us with the location address or website name immediately so that we can pursue a remedy.
Please contact us at <[email protected]> with a link to the suspected pirated material.
We appreciate your help in protecting our authors, and our ability to bring you valuable content.
You can contact us at <[email protected]> if you are having a problem with any aspect of the book, and we will do our best to address it.