Book Image

Hands-On Web Scraping with Python

By : Anish Chapagain
Book Image

Hands-On Web Scraping with Python

By: Anish Chapagain

Overview of this book

Web scraping is an essential technique used in many organizations to gather valuable data from web pages. This book will enable you to delve into web scraping techniques and methodologies. The book will introduce you to the fundamental concepts of web scraping techniques and how they can be applied to multiple sets of web pages. You'll use powerful libraries from the Python ecosystem such as Scrapy, lxml, pyquery, and bs4 to carry out web scraping operations. You will then get up to speed with simple to intermediate scraping operations such as identifying information from web pages and using patterns or attributes to retrieve information. This book adopts a practical approach to web scraping concepts and tools, guiding you through a series of use cases and showing you how to use the best tools and techniques to efficiently scrape web pages. You'll even cover the use of other popular web scraping tools, such as Selenium, Regex, and web-based APIs. By the end of this book, you will have learned how to efficiently scrape the web using different techniques with Python and other popular tools.

Related Content you might be interested in

Current Title:

Small book image
Hands-On Web Scraping with Python
Anish Chapagain
Jul, 2019 | 350 pages
Small book image
Python Web Scraping
Katharine Jarmul
May, 2017 | 220 pages
Small book image
R Web Scraping Quick Start Guide
Olgun Aydin
Oct, 2018 | 114 pages
Small book image
Python Web Scraping Cookbook
Michael Heydt
Feb, 2018 | 364 pages
Table of Contents (16 chapters)
Preface
Preface
Who this book is for
What this book covers
To get the most out of this book
Get in touch
Free Chapter
1
Section 1: Introduction to Web Scraping
Section 1: Introduction to Web Scraping
2
Web Scraping Fundamentals
Web Scraping Fundamentals
Introduction to web scraping
Understanding web development and technologies
Data finding techniques for the web
Summary
Further reading
3
Section 2: Beginning Web Scraping
Section 2: Beginning Web Scraping
4
Python and the Web – Using urllib and Requests
Python and the Web – Using urllib and Requests
Technical requirements
Accessing the web with Python
URL handling and operations with urllib and requests
Implementing HTTP methods
Summary
Further reading
5
Using LXML, XPath, and CSS Selectors
Using LXML, XPath, and CSS Selectors
Technical requirements
Introduction to XPath and CSS selector
Using web browser developer tools for accessing web content
Scraping using lxml, a Python library
Summary
Further reading
6
Scraping Using pyquery – a Python Library
Scraping Using pyquery – a Python Library
Technical requirements
Introduction to pyquery
Exploring pyquery
Web scraping using pyquery
Summary
Further reading
7
Web Scraping Using Scrapy and Beautiful Soup
Web Scraping Using Scrapy and Beautiful Soup
Technical requirements
Web scraping using Beautiful Soup
Web scraping using Scrapy
Deploying a web crawler
Summary
Further reading
8
Section 3: Advanced Concepts
Section 3: Advanced Concepts
9
Working with Secure Web
Working with Secure Web
Technical requirements
Introduction to secure web
HTML <form> processing
Handling user authentication
Working with cookies and sessions
Summary
Further reading
10
Data Extraction Using Web-Based APIs
Data Extraction Using Web-Based APIs
Technical requirements
Introduction to web APIs
Accessing web API and data formats
Web scraping using APIs
Summary
Further reading
11
Using Selenium to Scrape the Web
Using Selenium to Scrape the Web
Technical requirements
Introduction to Selenium
Using Selenium for web scraping
Summary
Further reading
12
Using Regex to Extract Data
Using Regex to Extract Data
Technical requirements
Overview of regular expressions
Using regular expressions to extract data
Summary
Further reading
13
Section 4: Conclusion
Section 4: Conclusion
14
Next Steps
Next Steps
Technical requirements
Managing scraped data
Analysis and visualization using pandas and matplotlib
Machine learning
Data mining
What's next?
Summary
Further reading
15
Other Books You May Enjoy
Other Books You May Enjoy
Leave a review - let other readers know what you think

Technical requirements

A web browser (Google Chrome or Mozilla Firefox) is required for this chapter, and we will be using the following Python libraries:

  • selenium (Python library)
  • re

If these libraries are not present in your current Python setup, then you can set them up or install them by referring to the Setting things up section in Chapter 2, Python and the Web – Using urllib and Requests.

In addition to the Python libraries and web browsers mentioned, we will be using WebDriver for Google Chrome.

Code files are available online at https://github.com/PacktPublishing/Hands-On-Web-Scraping-with-Python/tree/master/Chapter08.

Previous Section
End of Section 2
Next Section