Book Image

Python Web Scraping Cookbook

By : Michael Heydt
Book Image

Python Web Scraping Cookbook

By: Michael Heydt

Overview of this book

Python Web Scraping Cookbook is a solution-focused book that will teach you techniques to develop high-performance scrapers and deal with crawlers, sitemaps, forms automation, Ajax-based sites, caches, and more. You'll explore a number of real-world scenarios where every part of the development/product life cycle will be fully covered. You will not only develop the skills needed to design and develop reliable performance data flows, but also deploy your codebase to AWS. If you are involved in software engineering, product development, or data mining (or are interested in building data-driven products), you will find this book useful as each recipe has a clear purpose and objective. Right from extracting data from the websites to writing a sophisticated web crawler, the book's independent recipes will be a godsend. This book covers Python libraries, requests, and BeautifulSoup. You will learn about crawling, web spidering, working with Ajax websites, paginated items, and more. You will also learn to tackle problems such as 403 errors, working with proxy, scraping images, and LXML. By the end of this book, you will be able to scrape websites more efficiently and able to deploy and operate your scraper in the cloud.

Related Content you might be interested in

Current Title:

Small book image
Python Web Scraping Cookbook
Michael Heydt
Feb, 2018 | 364 pages
Small book image
Python Web Scraping
Katharine Jarmul
May, 2017 | 220 pages
Small book image
Hands-On Web Scraping with Python
Anish Chapagain
Jul, 2019 | 350 pages
Table of Contents (13 chapters)
Preface
Preface
Who this book is for
What this book covers
To get the most out of this book
Get in touch
Free Chapter
1
Getting Started with Scraping
Getting Started with Scraping
Introduction
Setting up a Python development environment
Scraping Python.org with Requests and Beautiful Soup
Scraping Python.org in urllib3 and Beautiful Soup
Scraping Python.org with Scrapy
Scraping Python.org with Selenium and PhantomJS
2
Data Acquisition and Extraction
Data Acquisition and Extraction
Introduction
How to parse websites and navigate the DOM using BeautifulSoup
Searching the DOM with Beautiful Soup's find methods
Querying the DOM with XPath and lxml
Querying data with XPath and CSS selectors
Using Scrapy selectors
Loading data in unicode / UTF-8
3
Processing Data
Processing Data
Introduction
Working with CSV and JSON data
Storing data using AWS S3
Storing data using MySQL
Storing data using PostgreSQL
Storing data in Elasticsearch
How to build robust ETL pipelines with AWS SQS
4
Working with Images, Audio, and other Assets
Working with Images, Audio, and other Assets
Introduction
Downloading media content from the web
Parsing a URL with urllib to get the filename
Determining the type of content for a URL
Determining the file extension from a content type
Downloading and saving images to the local file system
Downloading and saving images to S3
Generating thumbnails for images
Taking a screenshot of a website
Taking a screenshot of a website with an external service
Performing OCR on an image with pytesseract
Creating a Video Thumbnail
Ripping an MP4 video to an MP3
5
Scraping - Code of Conduct
Scraping - Code of Conduct
Introduction
Scraping legality and scraping politely
Respecting robots.txt
Crawling using the sitemap
Crawling with delays
Using identifiable user agents
Setting the number of concurrent requests per domain
Using auto throttling
Using an HTTP cache for development
6
Scraping Challenges and Solutions
Scraping Challenges and Solutions
Introduction
Retrying failed page downloads
Supporting page redirects
Waiting for content to be available in Selenium
Limiting crawling to a single domain
Processing infinitely scrolling pages
Controlling the depth of a crawl
Controlling the length of a crawl
Handling paginated websites
Handling forms and forms-based authorization
Handling basic authorization
Preventing bans by scraping via proxies
Randomizing user agents
Caching responses
7
Text Wrangling and Analysis
Text Wrangling and Analysis
Introduction
Installing NLTK
Performing sentence splitting
Performing tokenization
Performing stemming
Performing lemmatization
Determining and removing stop words
Calculating the frequency distributions of words
Identifying and removing rare words
Identifying and removing rare words
Removing punctuation marks
Piecing together n-grams
Scraping a job listing from StackOverflow
Reading and cleaning the description in the job listing
8
Searching, Mining and Visualizing Data
Searching, Mining and Visualizing Data
Introduction
Geocoding an IP address
How to collect IP addresses of Wikipedia edits
Visualizing contributor location frequency on Wikipedia
Creating a word cloud from a StackOverflow job listing
Crawling links on Wikipedia
Visualizing page relationships on Wikipedia
Calculating degrees of separation
9
Creating a Simple Data API
Creating a Simple Data API
Introduction
Creating a REST API with Flask-RESTful
Integrating the REST API with scraping code
Adding an API to find the skills for a job listing
Storing data in Elasticsearch as the result of a scraping request
Checking Elasticsearch for a listing before scraping
10
Creating Scraper Microservices with Docker
Creating Scraper Microservices with Docker
Introduction
Installing Docker
Installing a RabbitMQ container from Docker Hub
Running a Docker container (RabbitMQ)
Creating and running an Elasticsearch container
Stopping/restarting a container and removing the image
Creating a generic microservice with Nameko
Creating a scraping microservice
Creating a scraper container
Creating an API container
Composing and running the scraper locally with docker-compose
11
Making the Scraper as a Service Real
Making the Scraper as a Service Real
Introduction
Creating and configuring an Elastic Cloud trial account
Accessing the Elastic Cloud cluster with curl
Connecting to the Elastic Cloud cluster with Python
Performing an Elasticsearch query with the Python API
Using Elasticsearch to query for jobs with specific skills
Modifying the API to search for jobs by skill
Storing configuration in the environment
Creating an AWS IAM user and a key pair for ECS
Configuring Docker to authenticate with ECR
Pushing containers into ECR
Creating an ECS cluster
Creating a task to run our containers
Starting and accessing the containers in AWS
12
Other Books You May Enjoy
Other Books You May Enjoy
Leave a review - let other readers know what you think

Using an HTTP cache for development

The development of a web crawler is a process of exploration, and one that will iterate through various refinements to retrieve the requested information. During the development process, you will often be hitting remote servers, and the same URLs on those servers, over and over. This is not polite. Fortunately, scrapy also comes to the rescue by providing caching middleware that is specifically designed to help in this situation.

How to do it

Scrapy will cache requests using a middleware module named HttpCacheMiddleware. Enabling it is as simple as configuring the HTTPCACHE_ENABLED setting to True:

process = CrawlerProcess({
    'AUTOTHROTTLE_TARGET_CONCURRENCY': 3
})
process.crawl...
Previous Section
End of Chapter 5
Next Chapter