Anticipating Data Cleaning Issues When Importing Tabular Data with pandas

Book Overview & Buying
Table Of Contents

Python Data Cleaning Cookbook - Second Edition

By : Michael Walker

4.9 (22)

Buy this Book

Python Data Cleaning Cookbook

4.9 (22)

By: Michael Walker

Buy this Book

Overview of this book

Jumping into data analysis without proper data cleaning will certainly lead to incorrect results. The Python Data Cleaning Cookbook - Second Edition will show you tools and techniques for cleaning and handling data with Python for better outcomes. Fully updated to the latest version of Python and all relevant tools, this book will teach you how to manipulate and clean data to get it into a useful form. he current edition focuses on advanced techniques like machine learning and AI-specific approaches and tools for data cleaning along with the conventional ones. The book also delves into tips and techniques to process and clean data for ML, AI, and NLP models. You will learn how to filter and summarize data to gain insights and better understand what makes sense and what does not, along with discovering how to operate on data to address the issues you've identified. Next, you’ll cover recipes for using supervised learning and Naive Bayes analysis to identify unexpected values and classification errors and generate visualizations for exploratory data analysis (EDA) to identify unexpected values. Finally, you’ll build functions and classes that you can reuse without modification when you have new data. By the end of this Data Cleaning book, you'll know how to clean data and diagnose problems within it.

Preface

New in the Second Edition

Who this book is for

What this book covers

Get in touch

Free Chapter

Anticipating Data Cleaning Issues When Importing Tabular Data with pandas

Technical requirements

Importing CSV files

Importing Excel files

Importing data from SQL databases

Importing SPSS, Stata, and SAS data

Importing R data

Persisting tabular data

Summary

Anticipating Data Cleaning Issues When Working with HTML, JSON, and Spark Data

Technical requirements

Importing simple JSON data

Importing more complicated JSON data from an API

Importing data from web pages

Working with Spark data

Persisting JSON data

Versioning data

Summary

Taking the Measure of Your Data

Technical requirements

Getting a first look at your data

Selecting and organizing columns

Selecting rows

Generating frequencies for categorical variables

Generating summary statistics for continuous variables

Using generative AI to display descriptive statistics

Summary

Identifying Outliers in Subsets of Data

Technical requirements

Identifying outliers with one variable

Identifying outliers and unexpected values in bivariate relationships

Using subsetting to examine logical inconsistencies in variable relationships

Using linear regression to identify data points with significant influence

Using k-nearest neighbors to find outliers

Using Isolation Forest to find anomalies

Using PandasAI to identify outliers

Summary

Using Visualizations for the Identification of Unexpected Values

Technical requirements

Using histograms to examine the distribution of continuous variables

Using boxplots to identify outliers for continuous variables

Using grouped boxplots to uncover unexpected values in a particular group

Examining both distribution shape and outliers with violin plots

Using scatter plots to view bivariate relationships

Using line plots to examine trends in continuous variables

Generating a heat map based on a correlation matrix

Summary

Cleaning and Exploring Data with Series Operations

Technical requirements

Getting values from a pandas Series

Showing summary statistics for a pandas Series

Changing Series values

Changing Series values conditionally

Evaluating and cleaning string Series data

Working with dates

Using OpenAI for Series operations

Summary

Identifying and Fixing Missing Values

Technical requirements

Identifying missing values

Cleaning missing values

Imputing values with regression

Using k-nearest neighbors for imputation

Using random forest for imputation

Using PandasAI for imputation

Summary

Encoding, Transforming, and Scaling Features

Technical requirements

Creating training datasets and avoiding data leakage

Removing redundant or unhelpful features

Encoding categorical features: one-hot encoding

Encoding categorical features: ordinal encoding

Encoding categorical features with medium or high cardinality

Using mathematical transformations

Feature binning: equal width and equal frequency

k-means binning

Feature scaling

Summary

Fixing Messy Data When Aggregating

Technical requirements

Looping through data with itertuples (an anti-pattern)

Calculating summaries by group with NumPy arrays

Using groupby to organize data by groups

Using more complicated aggregation functions with groupby

Using user-defined functions and apply with groupby

Using groupby to change the unit of analysis of a DataFrame

Using pivot_table to change the unit of analysis of a DataFrame

Summary

Addressing Data Issues When Combining DataFrames

Technical requirements

Combining DataFrames vertically

Doing one-to-one merges

Doing one-to-one merges by multiple columns

Doing one-to-many merges

Doing many-to-many merges

Developing a merge routine

Summary

Tidying and Reshaping Data

Technical requirements

Removing duplicated rows

Fixing many-to-many relationships

Using stack and melt to reshape data from wide to long format

Melting multiple groups of columns

Using unstack and pivot to reshape data from long to wide format

Summary

Automate Data Cleaning with User-Defined Functions, Classes, and Pipelines

Technical requirements

Functions for getting a first look at our data

Functions for displaying summary statistics and frequencies

Functions for identifying outliers and unexpected values

Functions for aggregating or combining data

Classes that contain the logic for updating Series values

Classes that handle non-tabular data structures

Functions for checking overall data quality

Pre-processing data with pipelines: a simple example

Pre-processing data with pipelines: a more complicated example

Summary

Index

Python Data Cleaning Cookbook - Second Edition

By : Michael Walker

Python Data Cleaning Cookbook

By: Michael Walker

Overview of this book

Anticipating Data Cleaning Issues When Importing Tabular Data with pandas

Confirmation

Buy this book with your credits?

Submit Your Feedback

Create a Free Account To Continue Reading

Sign in to activate your 7-day free access