Book Image

Python Web Scraping Cookbook

By : Michael Heydt

Book Image

Python Web Scraping Cookbook

By: Michael Heydt

Overview of this book

Python Web Scraping Cookbook is a solution-focused book that will teach you techniques to develop high-performance scrapers and deal with crawlers, sitemaps, forms automation, Ajax-based sites, caches, and more. You'll explore a number of real-world scenarios where every part of the development/product life cycle will be fully covered. You will not only develop the skills needed to design and develop reliable performance data flows, but also deploy your codebase to AWS. If you are involved in software engineering, product development, or data mining (or are interested in building data-driven products), you will find this book useful as each recipe has a clear purpose and objective. Right from extracting data from the websites to writing a sophisticated web crawler, the book's independent recipes will be a godsend. This book covers Python libraries, requests, and BeautifulSoup. You will learn about crawling, web spidering, working with Ajax websites, paginated items, and more. You will also learn to tackle problems such as 403 errors, working with proxy, scraping images, and LXML. By the end of this book, you will be able to scrape websites more efficiently and able to deploy and operate your scraper in the cloud.

Preface

Who this book is for

What this book covers

To get the most out of this book

Free Chapter

Getting Started with Scraping

Getting Started with Scraping

Setting up a Python development environment

Scraping Python.org with Requests and Beautiful Soup

Scraping Python.org in urllib3 and Beautiful Soup

Scraping Python.org with Scrapy

Scraping Python.org with Selenium and PhantomJS

Data Acquisition and Extraction

Data Acquisition and Extraction

How to parse websites and navigate the DOM using BeautifulSoup

Searching the DOM with Beautiful Soup's find methods

Querying the DOM with XPath and lxml

Querying data with XPath and CSS selectors

Using Scrapy selectors

Loading data in unicode / UTF-8

Processing Data

Processing Data

Working with CSV and JSON data

Storing data using AWS S3

Storing data using MySQL

Storing data using PostgreSQL

Storing data in Elasticsearch

How to build robust ETL pipelines with AWS SQS

Working with Images, Audio, and other Assets

Working with Images, Audio, and other Assets

Downloading media content from the web

Parsing a URL with urllib to get the filename

Determining the type of content for a URL

Determining the file extension from a content type

Downloading and saving images to the local file system

Downloading and saving images to S3

Generating thumbnails for images

Taking a screenshot of a website

Taking a screenshot of a website with an external service

Performing OCR on an image with pytesseract

Creating a Video Thumbnail

Ripping an MP4 video to an MP3

Scraping - Code of Conduct

Scraping - Code of Conduct

Scraping legality and scraping politely

Respecting robots.txt

Crawling using the sitemap

Crawling with delays

Using identifiable user agents

Setting the number of concurrent requests per domain

Using auto throttling

Using an HTTP cache for development

Scraping Challenges and Solutions

Scraping Challenges and Solutions

Retrying failed page downloads

Supporting page redirects

Waiting for content to be available in Selenium

Limiting crawling to a single domain

Processing infinitely scrolling pages

Controlling the depth of a crawl

Controlling the length of a crawl

Handling paginated websites

Handling forms and forms-based authorization

Handling basic authorization

Preventing bans by scraping via proxies

Randomizing user agents

Caching responses

Text Wrangling and Analysis

Text Wrangling and Analysis

Installing NLTK

Performing sentence splitting

Performing tokenization

Performing stemming

Performing lemmatization

Determining and removing stop words

Calculating the frequency distributions of words

Identifying and removing rare words

Identifying and removing rare words

Removing punctuation marks

Piecing together n-grams

Scraping a job listing from StackOverflow

Reading and cleaning the description in the job listing

Searching, Mining and Visualizing Data

Searching, Mining and Visualizing Data

Geocoding an IP address

How to collect IP addresses of Wikipedia edits

Visualizing contributor location frequency on Wikipedia

Creating a word cloud from a StackOverflow job listing

Crawling links on Wikipedia

Visualizing page relationships on Wikipedia

Calculating degrees of separation

Creating a Simple Data API

Creating a Simple Data API

Creating a REST API with Flask-RESTful

Integrating the REST API with scraping code

Adding an API to find the skills for a job listing

Storing data in Elasticsearch as the result of a scraping request

Checking Elasticsearch for a listing before scraping

Creating Scraper Microservices with Docker

Creating Scraper Microservices with Docker

Installing Docker

Installing a RabbitMQ container from Docker Hub

Running a Docker container (RabbitMQ)

Creating and running an Elasticsearch container

Stopping/restarting a container and removing the image

Creating a generic microservice with Nameko

Creating a scraping microservice

Creating a scraper container

Creating an API container

Composing and running the scraper locally with docker-compose

Making the Scraper as a Service Real

Making the Scraper as a Service Real

Creating and configuring an Elastic Cloud trial account

Accessing the Elastic Cloud cluster with curl

Connecting to the Elastic Cloud cluster with Python

Performing an Elasticsearch query with the Python API

Using Elasticsearch to query for jobs with specific skills

Modifying the API to search for jobs by skill

Storing configuration in the environment

Creating an AWS IAM user and a key pair for ECS

Configuring Docker to authenticate with ECR

Pushing containers into ECR

Creating an ECS cluster

Creating a task to run our containers

Starting and accessing the containers in AWS

Other Books You May Enjoy

Other Books You May Enjoy

Leave a review - let other readers know what you think

Customer Reviews

5 star

0

4 star

0

3 star

0

2 star

0

1 star

0

Using identifiable user agents

What happens if you violate the terms of service and get flagged by the website owner? How can you help the site owners in contacting you, so that they can nicely ask you to back off to what they consider a reasonable level of scraping?

What you can do to facilitate this is add info about yourself in the User-Agent header of the requests. We have seen an example of this in robots.txt files, such as from amazon.com. In their robots.txt is an explicit statement of a user agent for Google: GoogleBot.

During scraping, you can embed your own information within the User-Agent header of the HTTP requests. To be polite, you can enter something such as 'MyCompany-MyCrawler ([email protected])'. The remote server, if tagging you in violation, will definitely be capturing this information, and if provided like this, it gives them a convenient means...