Book Image

Python Data Cleaning Cookbook

By : Michael Walker
Book Image

Python Data Cleaning Cookbook

By: Michael Walker

Overview of this book

Getting clean data to reveal insights is essential, as directly jumping into data analysis without proper data cleaning may lead to incorrect results. This book shows you tools and techniques that you can apply to clean and handle data with Python. You'll begin by getting familiar with the shape of data by using practices that can be deployed routinely with most data sources. Then, the book teaches you how to manipulate data to get it into a useful form. You'll also learn how to filter and summarize data to gain insights and better understand what makes sense and what does not, along with discovering how to operate on data to address the issues you've identified. Moving on, you'll perform key tasks, such as handling missing values, validating errors, removing duplicate data, monitoring high volumes of data, and handling outliers and invalid dates. Next, you'll cover recipes on using supervised learning and Naive Bayes analysis to identify unexpected values and classification errors, and generate visualizations for exploratory data analysis (EDA) to visualize unexpected values. Finally, you'll build functions and classes that you can reuse without modification when you have new data. By the end of this Python book, you'll be equipped with all the key skills that you need to clean data and diagnose problems within it.

Related Content you might be interested in

Current Title:

Small book image
Python Data Cleaning Cookbook
Michael Walker
Dec, 2020 | 436 pages
Small book image
Data Cleaning and Exploration with Machine Learning
Michael Walker
Aug, 2022 | 542 pages
Small book image
Mastering Exploratory Analysis with pandas
Harish Garg
Sep, 2018 | 140 pages
Small book image
Pandas Cookbook
Theodore Petrou
Oct, 2017 | 532 pages
Table of Contents (12 chapters)
Preface
Preface
Who this book is for
What this book covers
To get the most out of this book
Download the color images
Conventions used
Sections
Get in touch
Reviews
1
Chapter 1: Anticipating Data Cleaning Issues when Importing Tabular Data into pandas
Chapter 1: Anticipating Data Cleaning Issues when Importing Tabular Data into pandas
Technical requirements
Importing CSV files
Importing Excel files
Importing data from SQL databases
Importing SPSS, Stata, and SAS data
Importing R data
Persisting tabular data
Free Chapter
2
Chapter 2: Anticipating Data Cleaning Issues when Importing HTML and JSON into pandas
Chapter 2: Anticipating Data Cleaning Issues when Importing HTML and JSON into pandas
Technical requirements
Importing simple JSON data
Importing more complicated JSON data from an API
Importing data from web pages
Persisting JSON data
3
Chapter 3: Taking the Measure of Your Data
Chapter 3: Taking the Measure of Your Data
Technical requirements
Getting a first look at your data
Selecting and organizing columns
Selecting rows
Generating frequencies for categorical variables
Generating summary statistics for continuous variables
4
Chapter 4: Identifying Missing Values and Outliers in Subsets of Data
Chapter 4: Identifying Missing Values and Outliers in Subsets of Data
Technical requirements
Finding missing values
Identifying outliers with one variable
Identifying outliers and unexpected values in bivariate relationships
Using subsetting to examine logical inconsistencies in variable relationships
Using linear regression to identify data points with significant influence
Using k-nearest neighbor to find outliers
Using Isolation Forest to find anomalies
5
Chapter 5: Using Visualizations for the Identification of Unexpected Values
Chapter 5: Using Visualizations for the Identification of Unexpected Values
Technical requirements
Using histograms to examine the distribution of continuous variables
Using boxplots to identify outliers for continuous variables
Using grouped boxplots to uncover unexpected values in a particular group
Examining both the distribution shape and outliers with violin plots
Using scatter plots to view bivariate relationships
Using line plots to examine trends in continuous variables
Generating a heat map based on a correlation matrix
6
Chapter 6: Cleaning and Exploring Data with Series Operations
Chapter 6: Cleaning and Exploring Data with Series Operations
Technical requirements
Getting values from a pandas series
Showing summary statistics for a pandas series
Changing series values
Changing series values conditionally
Evaluating and cleaning string series data
Working with dates
Identifying and cleaning missing data
Missing value imputation with K-nearest neighbor
7
Chapter 7: Fixing Messy Data when Aggregating
Chapter 7: Fixing Messy Data when Aggregating
Technical requirements
Looping through data with itertuples (an anti-pattern)
Calculating summaries by group with NumPy arrays
Using groupby to organize data by groups
Using more complicated aggregation functions with groupby
Using user-defined functions and apply with groupby
Using groupby to change the unit of analysis of a DataFrame
8
Chapter 8: Addressing Data Issues When Combining DataFrames
Chapter 8: Addressing Data Issues When Combining DataFrames
Technical requirements
Combining DataFrames vertically
Doing one-to-one merges
Using multiple merge-by columns
Doing one-to-many merges
Doing many-to-many merges
Developing a merge routine
9
Chapter 9: Tidying and Reshaping Data
Chapter 9: Tidying and Reshaping Data
Technical requirements
Removing duplicated rows
Fixing many-to-many relationships
Using stack and melt to reshape data from wide to long format
Melting multiple groups of columns
Using unstack and pivot to reshape data from long to wide
10
Chapter 10: User-Defined Functions and Classes to Automate Data Cleaning
Chapter 10: User-Defined Functions and Classes to Automate Data Cleaning
Technical requirements
Functions for getting a first look at our data
Functions for displaying summary statistics and frequencies
Functions for identifying outliers and unexpected values
Functions for aggregating or combining data
Classes that contain the logic for updating series values
Classes that handle non-tabular data structures
11
Other Books You May Enjoy
Other Books You May Enjoy
Leave a review - let other readers know what you think

What this book covers

Chapter 1, Anticipating Data Cleaning Issues when Importing Tabular Data into pandas, explores tools for loading CSV files, Excel files, relational database tables, SAS, SPSS, and Stata files, and R files into pandas DataFrames.

Chapter 2, Anticipating Data Cleaning Issues when Importing HTML and JSON into pandas, discusses techniques for reading and normalizing JSON data, and for web scraping.

Chapter 3, Taking the Measure of Your Data, introduces common techniques for navigating around a DataFrame, selecting columns and rows, and generating summary statistics.

Chapter 4, Identifying Missing Values and Outliers in Subsets of Data, explores a wide range of strategies to identify missing values and outliers across a whole DataFrame and by selected groups.

Chapter 5, Using Visualizations for the Identification of Unexpected Values, demonstrates the use of matplotlib and seaborn tools to visualize how key variables are distributed, including with histograms, boxplots, scatter plots, line plots, and violin plots.

Chapter 6, Cleaning and Exploring Data with Series Operations, discusses updating pandas series with scalars, arithmetic operations, and conditional statements based on the values of one or more series.

Chapter 7, Fixing Messy Data when Aggregating, demonstrates multiple approaches to aggregating data by group, and discusses when to choose one approach over the others.

Chapter 8, Addressing Data Issues when Combining DataFrames, examines different strategies for concatenating and merging data, and how to anticipate common data challenges when combining data.

Chapter 9, Tidying and Reshaping Data, introduces several strategies for de-duplicating, stacking, melting, and pivoting data.

Chapter 10, User-Defined Functions and Classes to Automate Data Cleaning, examines how to turn many of the techniques from the first nine chapters into reusable code.

Previous Section
Next Section