Sign In Start Free Trial

Book Overview & Buying
Table Of Contents

Python Web Scraping Cookbook

By : Michael Heydt

2.3 (3)

Python Web Scraping Cookbook

2.3 (3)

By: Michael Heydt

Overview of this book

Python Web Scraping Cookbook is a solution-focused book that will teach you techniques to develop high-performance scrapers and deal with crawlers, sitemaps, forms automation, Ajax-based sites, caches, and more. You'll explore a number of real-world scenarios where every part of the development/product life cycle will be fully covered. You will not only develop the skills needed to design and develop reliable performance data flows, but also deploy your codebase to AWS. If you are involved in software engineering, product development, or data mining (or are interested in building data-driven products), you will find this book useful as each recipe has a clear purpose and objective. Right from extracting data from the websites to writing a sophisticated web crawler, the book's independent recipes will be a godsend. This book covers Python libraries, requests, and BeautifulSoup. You will learn about crawling, web spidering, working with Ajax websites, paginated items, and more. You will also learn to tackle problems such as 403 errors, working with proxy, scraping images, and LXML. By the end of this book, you will be able to scrape websites more efficiently and able to deploy and operate your scraper in the cloud.

Preface

Preface

Who this book is for

What this book covers

To get the most out of this book

Get in touch

Free Chapter

Getting Started with Scraping

Getting Started with Scraping

Introduction

Setting up a Python development environment

Scraping Python.org with Requests and Beautiful Soup

Scraping Python.org in urllib3 and Beautiful Soup

Scraping Python.org with Scrapy

Scraping Python.org with Selenium and PhantomJS

Data Acquisition and Extraction

Data Acquisition and Extraction

Introduction

How to parse websites and navigate the DOM using BeautifulSoup

Searching the DOM with Beautiful Soup's find methods

Querying the DOM with XPath and lxml

Querying data with XPath and CSS selectors

Using Scrapy selectors

Loading data in unicode / UTF-8

Processing Data

Processing Data

Introduction

Working with CSV and JSON data

Storing data using AWS S3

Storing data using MySQL

Storing data using PostgreSQL

Storing data in Elasticsearch

How to build robust ETL pipelines with AWS SQS

Working with Images, Audio, and other Assets

Working with Images, Audio, and other Assets

Introduction

Downloading media content from the web

Parsing a URL with urllib to get the filename

Determining the type of content for a URL

Determining the file extension from a content type

Downloading and saving images to the local file system

Downloading and saving images to S3

Generating thumbnails for images

Taking a screenshot of a website

Taking a screenshot of a website with an external service

Performing OCR on an image with pytesseract

Creating a Video Thumbnail

Ripping an MP4 video to an MP3

Scraping - Code of Conduct

Scraping - Code of Conduct

Introduction

Scraping legality and scraping politely

Respecting robots.txt

Crawling using the sitemap

Crawling with delays

Using identifiable user agents

Setting the number of concurrent requests per domain

Using auto throttling

Using an HTTP cache for development

Scraping Challenges and Solutions

Scraping Challenges and Solutions

Introduction

Retrying failed page downloads

Supporting page redirects

Waiting for content to be available in Selenium

Limiting crawling to a single domain

Processing infinitely scrolling pages

Controlling the depth of a crawl

Controlling the length of a crawl

Handling paginated websites

Handling forms and forms-based authorization

Handling basic authorization

Preventing bans by scraping via proxies

Randomizing user agents

Caching responses

Text Wrangling and Analysis

Text Wrangling and Analysis

Introduction

Installing NLTK

Performing sentence splitting

Performing tokenization

Performing stemming

Performing lemmatization

Determining and removing stop words

Calculating the frequency distributions of words

Identifying and removing rare words

Identifying and removing rare words

Removing punctuation marks

Piecing together n-grams

Scraping a job listing from StackOverflow

Reading and cleaning the description in the job listing

Searching, Mining and Visualizing Data

Searching, Mining and Visualizing Data

Introduction

Geocoding an IP address

How to collect IP addresses of Wikipedia edits

Visualizing contributor location frequency on Wikipedia

Creating a word cloud from a StackOverflow job listing

Crawling links on Wikipedia

Visualizing page relationships on Wikipedia

Calculating degrees of separation

Creating a Simple Data API

Creating a Simple Data API

Introduction

Creating a REST API with Flask-RESTful

Integrating the REST API with scraping code

Adding an API to find the skills for a job listing

Storing data in Elasticsearch as the result of a scraping request

Checking Elasticsearch for a listing before scraping

Creating Scraper Microservices with Docker

Creating Scraper Microservices with Docker

Introduction

Installing Docker

Installing a RabbitMQ container from Docker Hub

Running a Docker container (RabbitMQ)

Creating and running an Elasticsearch container

Stopping/restarting a container and removing the image

Creating a generic microservice with Nameko

Creating a scraping microservice

Creating a scraper container

Creating an API container

Composing and running the scraper locally with docker-compose

Making the Scraper as a Service Real

Making the Scraper as a Service Real

Introduction

Creating and configuring an Elastic Cloud trial account

Accessing the Elastic Cloud cluster with curl

Connecting to the Elastic Cloud cluster with Python

Performing an Elasticsearch query with the Python API

Using Elasticsearch to query for jobs with specific skills

Modifying the API to search for jobs by skill

Storing configuration in the environment

Creating an AWS IAM user and a key pair for ECS

Configuring Docker to authenticate with ECR

Pushing containers into ECR

Creating an ECS cluster

Creating a task to run our containers

Starting and accessing the containers in AWS

Other Books You May Enjoy

Other Books You May Enjoy

Leave a review - let other readers know what you think

Performing tokenization

Tokenization is the process of converting text into tokens. These tokens can be paragraphs, sentences, and common individual words, and are commonly based at the word level. NLTK comes with a number of tokenizers that will be demonstrated in this recipe.

How to do it

The code for this example is in the 07/02_tokenize.py file. This extends the sentence splitter to demonstrate five different tokenization techniques. The first sentence in the file will be the only one tokenized so that we keep the amount of output to a reasonable amount:

The first step is to simply use the built-in Python string .split() method. This results in the following:

print(first_sentence.split())
['We', 'are...

CONTINUE READING

83

Tech Concepts

36

Programming languages

73

Tech Tools

Unlimited access to the largest independent learning library in tech of over 8,000 expert-authored tech books and videos.

Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.

50+ new titles added per month and exclusive early access to books as they are being written.

Python Web Scraping Cookbook

Search

Your notes and bookmarks