Book Image

Pandas Cookbook

By : Theodore Petrou
Book Image

Pandas Cookbook

By: Theodore Petrou

Overview of this book

This book will provide you with unique, idiomatic, and fun recipes for both fundamental and advanced data manipulation tasks with pandas 0.20. Some recipes focus on achieving a deeper understanding of basic principles, or comparing and contrasting two similar operations. Other recipes will dive deep into a particular dataset, uncovering new and unexpected insights along the way. The pandas library is massive, and it's common for frequent users to be unaware of many of its more impressive features. The official pandas documentation, while thorough, does not contain many useful examples of how to piece together multiple commands like one would do during an actual analysis. This book guides you, as if you were looking over the shoulder of an expert, through practical situations that you are highly likely to encounter. Many advanced recipes combine several different features across the pandas 0.20 library to generate results.

Related Content you might be interested in

Current Title:

Small book image
Pandas Cookbook
Theodore Petrou
Oct, 2017 | 532 pages
Small book image
Learning pandas
Michael Heydt
Jun, 2017 | 446 pages
Small book image
Mastering Exploratory Analysis with pandas
Harish Garg
Sep, 2018 | 140 pages
Small book image
Python Data Cleaning Cookbook
Michael Walker
Dec, 2020 | 436 pages
Table of Contents (12 chapters)
Preface
Preface
What this book covers
What you need for this book
Who this book is for
Conventions
Assumptions for every recipe
Dataset Descriptions
Sections
Reader feedback
Customer support
Free Chapter
1
Pandas Foundations
Pandas Foundations
Introduction
Dissecting the anatomy of a DataFrame
Accessing the main DataFrame components
Understanding data types
Selecting a single column of data as a Series
Calling Series methods
Working with operators on a Series
Chaining Series methods together
Making the index meaningful
Renaming row and column names
Creating and deleting columns
2
Essential DataFrame Operations
Essential DataFrame Operations
Introduction
Selecting multiple DataFrame columns
Selecting columns with methods
Ordering column names sensibly
Operating on the entire DataFrame
Chaining DataFrame methods together
Working with operators on a DataFrame
Comparing missing values
Transposing the direction of a DataFrame operation
Determining college campus diversity
3
Beginning Data Analysis
Beginning Data Analysis
Introduction
Developing a data analysis routine
Reducing memory by changing data types
Selecting the smallest of the largest
Selecting the largest of each group by sorting
Replicating nlargest with sort_values
Calculating a trailing stop order price
4
Selecting Subsets of Data
Selecting Subsets of Data
Introduction
Selecting Series data
Selecting DataFrame rows
Selecting DataFrame rows and columns simultaneously
Selecting data with both integers and labels
Speeding up scalar selection
Slicing rows lazily
Slicing lexicographically
5
Boolean Indexing
Boolean Indexing
Introduction
Calculating boolean statistics
Constructing multiple boolean conditions
Filtering with boolean indexing
Replicating boolean indexing with index selection
Selecting with unique and sorted indexes
Gaining perspective on stock prices
Translating SQL WHERE clauses
Determining the normality of stock market returns
Improving readability of boolean indexing with the query method
Preserving Series with the where method
Masking DataFrame rows
Selecting with booleans, integer location, and labels
6
Index Alignment
Index Alignment
Introduction
Examining the Index object
Producing Cartesian products
Exploding indexes
Filling values with unequal indexes
Appending columns from different DataFrames
Highlighting the maximum value from each column
Replicating idxmax with method chaining
Finding the most common maximum
7
Grouping for Aggregation, Filtration, and Transformation
Grouping for Aggregation, Filtration, and Transformation
Introduction
Defining an aggregation
Grouping and aggregating with multiple columns and functions
Removing the MultiIndex after grouping
Customizing an aggregation function
Customizing aggregating functions with *args and **kwargs
Examining the groupby object
Filtering for states with a minority majority
Transforming through a weight loss bet
Calculating weighted mean SAT scores per state with apply
Grouping by continuous variables
Counting the total number of flights between cities
Finding the longest streak of on-time flights
8
Restructuring Data into a Tidy Form
Restructuring Data into a Tidy Form
Introduction
Tidying variable values as column names with stack
Tidying variable values as column names with melt
Stacking multiple groups of variables simultaneously
Inverting stacked data
Unstacking after a groupby aggregation
Replicating pivot_table with a groupby aggregation
Renaming axis levels for easy reshaping
Tidying when multiple variables are stored as column names
Tidying when multiple variables are stored as column values
Tidying when two or more values are stored in the same cell
Tidying when variables are stored in column names and values
Tidying when multiple observational units are stored in the same table
9
Combining Pandas Objects
Combining Pandas Objects
Introduction
Appending new rows to DataFrames
Concatenating multiple DataFrames together
Comparing President Trump's and Obama's approval ratings
Understanding the differences between concat, join, and merge
Connecting to SQL databases
10
Time Series Analysis
Time Series Analysis
Introduction
Understanding the difference between Python and pandas date tools
Slicing time series intelligently
Using methods that only work with a DatetimeIndex
Counting the number of weekly crimes
Aggregating weekly crime and traffic accidents separately
Measuring crime by weekday and year
Grouping with anonymous functions with a DatetimeIndex
Grouping by a Timestamp and another column
Finding the last time crime was 20% lower with merge_asof
11
Visualization with Matplotlib, Pandas, and Seaborn
Visualization with Matplotlib, Pandas, and Seaborn
Introduction
Getting started with matplotlib
Visualizing data with matplotlib
Plotting basics with pandas
Visualizing the flights dataset
Stacking area charts to discover emerging trends
Understanding the differences between seaborn and pandas
Doing multivariate analysis with seaborn Grids
Uncovering Simpson's paradox in the diamonds dataset with seaborn

Tidying variable values as column names with stack

To help understand the differences between tidy and messy data, let's take a look at a simple table that may or may not be in tidy form:

>>> state_fruit = pd.read_csv('data/state_fruit.csv', index_col=0)
>>> state_fruit

There does not appear to be anything messy about this table, and the information is easily consumable. However, according to the tidy principles, it isn't actually tidy. Each column name is actually the value of a variable. In fact, none of the variable names are even present in the DataFrame. One of the first steps to transform a messy dataset into tidy data is to identify all of the variables. In this particular dataset, we have variables for state and fruit. There's also the numeric data that wasn't identified anywhere in the context of the problem. We can label...

Previous Section
End of Section 3
Next Section