Book Image

Web Scraping with Python

By : Richard Penman
Book Image

Web Scraping with Python

By: Richard Penman

Overview of this book

Related Content you might be interested in

No titles found
Table of Contents (16 chapters)
Web Scraping with Python
Web Scraping with Python
Credits
Credits
About the Author
About the Author
About the Reviewers
About the Reviewers
www.PacktPub.com
www.PacktPub.com
Preface
Preface
Free Chapter
1
Introduction to Web Scraping
Introduction to Web Scraping
When is web scraping useful?
Is web scraping legal?
Background research
Crawling your first website
Summary
2
Scraping the Data
Scraping the Data
Analyzing a web page
Three approaches to scrape a web page
Summary
3
Caching Downloads
Caching Downloads
Adding cache support to the link crawler
Disk cache
Database cache
Summary
4
Concurrent Downloading
Concurrent Downloading
One million web pages
Sequential crawler
Threaded crawler
Performance
Summary
5
Dynamic Content
Dynamic Content
An example dynamic web page
Reverse engineering a dynamic web page
Rendering a dynamic web page
Summary
6
Interacting with Forms
Interacting with Forms
The Login form
Extending the login script to update content
Automating forms with the Mechanize module
Summary
7
Solving CAPTCHA
Solving CAPTCHA
Registering an account
Optical Character Recognition
Solving complex CAPTCHAs
Summary
8
Scrapy
Scrapy
Installation
Starting a project
Visual scraping with Portia
Automated scraping with Scrapely
Summary
9
Overview
Overview
Google search engine
Facebook
Gap
BMW
Summary
Index
Index

Chapter 1. Introduction to Web Scraping

In this chapter, we will cover the following topics:

  • Introduce the field of web scraping

  • Explain the legal challenges

  • Perform background research on our target website

  • Progressively building our own advanced web crawler

Previous Section
End of Section 1
Next Section