Book Image

Pandas Cookbook

By : Theodore Petrou
Book Image

Pandas Cookbook

By: Theodore Petrou

Overview of this book

This book will provide you with unique, idiomatic, and fun recipes for both fundamental and advanced data manipulation tasks with pandas 0.20. Some recipes focus on achieving a deeper understanding of basic principles, or comparing and contrasting two similar operations. Other recipes will dive deep into a particular dataset, uncovering new and unexpected insights along the way. The pandas library is massive, and it's common for frequent users to be unaware of many of its more impressive features. The official pandas documentation, while thorough, does not contain many useful examples of how to piece together multiple commands like one would do during an actual analysis. This book guides you, as if you were looking over the shoulder of an expert, through practical situations that you are highly likely to encounter. Many advanced recipes combine several different features across the pandas 0.20 library to generate results.

Related Content you might be interested in

Current Title:

Small book image
Pandas Cookbook
Theodore Petrou
Oct, 2017 | 532 pages
Small book image
Learning pandas
Michael Heydt
Jun, 2017 | 446 pages
Small book image
Mastering Exploratory Analysis with pandas
Harish Garg
Sep, 2018 | 140 pages
Small book image
Python Data Cleaning Cookbook
Michael Walker
Dec, 2020 | 436 pages
Table of Contents (12 chapters)
Preface
Preface
What this book covers
What you need for this book
Who this book is for
Conventions
Assumptions for every recipe
Dataset Descriptions
Sections
Reader feedback
Customer support
Free Chapter
1
Pandas Foundations
Pandas Foundations
Introduction
Dissecting the anatomy of a DataFrame
Accessing the main DataFrame components
Understanding data types
Selecting a single column of data as a Series
Calling Series methods
Working with operators on a Series
Chaining Series methods together
Making the index meaningful
Renaming row and column names
Creating and deleting columns
2
Essential DataFrame Operations
Essential DataFrame Operations
Introduction
Selecting multiple DataFrame columns
Selecting columns with methods
Ordering column names sensibly
Operating on the entire DataFrame
Chaining DataFrame methods together
Working with operators on a DataFrame
Comparing missing values
Transposing the direction of a DataFrame operation
Determining college campus diversity
3
Beginning Data Analysis
Beginning Data Analysis
Introduction
Developing a data analysis routine
Reducing memory by changing data types
Selecting the smallest of the largest
Selecting the largest of each group by sorting
Replicating nlargest with sort_values
Calculating a trailing stop order price
4
Selecting Subsets of Data
Selecting Subsets of Data
Introduction
Selecting Series data
Selecting DataFrame rows
Selecting DataFrame rows and columns simultaneously
Selecting data with both integers and labels
Speeding up scalar selection
Slicing rows lazily
Slicing lexicographically
5
Boolean Indexing
Boolean Indexing
Introduction
Calculating boolean statistics
Constructing multiple boolean conditions
Filtering with boolean indexing
Replicating boolean indexing with index selection
Selecting with unique and sorted indexes
Gaining perspective on stock prices
Translating SQL WHERE clauses
Determining the normality of stock market returns
Improving readability of boolean indexing with the query method
Preserving Series with the where method
Masking DataFrame rows
Selecting with booleans, integer location, and labels
6
Index Alignment
Index Alignment
Introduction
Examining the Index object
Producing Cartesian products
Exploding indexes
Filling values with unequal indexes
Appending columns from different DataFrames
Highlighting the maximum value from each column
Replicating idxmax with method chaining
Finding the most common maximum
7
Grouping for Aggregation, Filtration, and Transformation
Grouping for Aggregation, Filtration, and Transformation
Introduction
Defining an aggregation
Grouping and aggregating with multiple columns and functions
Removing the MultiIndex after grouping
Customizing an aggregation function
Customizing aggregating functions with *args and **kwargs
Examining the groupby object
Filtering for states with a minority majority
Transforming through a weight loss bet
Calculating weighted mean SAT scores per state with apply
Grouping by continuous variables
Counting the total number of flights between cities
Finding the longest streak of on-time flights
8
Restructuring Data into a Tidy Form
Restructuring Data into a Tidy Form
Introduction
Tidying variable values as column names with stack
Tidying variable values as column names with melt
Stacking multiple groups of variables simultaneously
Inverting stacked data
Unstacking after a groupby aggregation
Replicating pivot_table with a groupby aggregation
Renaming axis levels for easy reshaping
Tidying when multiple variables are stored as column names
Tidying when multiple variables are stored as column values
Tidying when two or more values are stored in the same cell
Tidying when variables are stored in column names and values
Tidying when multiple observational units are stored in the same table
9
Combining Pandas Objects
Combining Pandas Objects
Introduction
Appending new rows to DataFrames
Concatenating multiple DataFrames together
Comparing President Trump's and Obama's approval ratings
Understanding the differences between concat, join, and merge
Connecting to SQL databases
10
Time Series Analysis
Time Series Analysis
Introduction
Understanding the difference between Python and pandas date tools
Slicing time series intelligently
Using methods that only work with a DatetimeIndex
Counting the number of weekly crimes
Aggregating weekly crime and traffic accidents separately
Measuring crime by weekday and year
Grouping with anonymous functions with a DatetimeIndex
Grouping by a Timestamp and another column
Finding the last time crime was 20% lower with merge_asof
11
Visualization with Matplotlib, Pandas, and Seaborn
Visualization with Matplotlib, Pandas, and Seaborn
Introduction
Getting started with matplotlib
Visualizing data with matplotlib
Plotting basics with pandas
Visualizing the flights dataset
Stacking area charts to discover emerging trends
Understanding the differences between seaborn and pandas
Doing multivariate analysis with seaborn Grids
Uncovering Simpson's paradox in the diamonds dataset with seaborn

Dissecting the anatomy of a DataFrame

Before diving deep into pandas, it is worth knowing the components of the DataFrame. Visually, the outputted display of a pandas DataFrame (in a Jupyter Notebook) appears to be nothing more than an ordinary table of data consisting of rows and columns. Hiding beneath the surface are the three components--the index, columns, and data (also known as values) that you must be aware of in order to maximize the DataFrame's full potential.

Getting ready

This recipe reads in the movie dataset into a pandas DataFrame and provides a labeled diagram of all its major components.

How to do it...

  1. Use the read_csv function to read in the movie dataset, and display the first five rows with the head method:
>>> movie = pd.read_csv('data/movie.csv')
>>> movie.head()
  1. Analyze the labeled anatomy of the DataFrame:

How it works...

Pandas first reads the data from disk into memory and into a DataFrame using the excellent and versatile read_csv function. The output for both the columns and the index is in bold font, which makes them easy to identify. By convention, the terms index label and column name refer to the individual members of the index and columns, respectively. The term index refers to all the index labels as a whole just as the term columns refers to all the column names as a whole.

The columns and the index serve a particular purpose, and that is to provide labels for the columns and rows of the DataFrame. These labels allow for direct and easy access to different subsets of data. When multiple Series or DataFrames are combined, the indexes align first before any calculation occurs. Collectively, the columns and the index are known as the axes.

A DataFrame has two axes--a vertical axis (the index) and a horizontal axis(the columns). Pandas borrows convention from NumPy and uses the integers 0/1 as another way of referring to the vertical/horizontal axis.

DataFrame data (values) is always in regular font and is an entirely separate component from the columns or index. Pandas uses NaN (not a number) to represent missing values. Notice that even though the color column has only string values, it uses NaN to represent a missing value.

The three consecutive dots in the middle of the columns indicate that there is at least one column that exists but is not displayed due to the number of columns exceeding the predefined display limits.

The Python standard library contains the csv module, which can be used to parse and read in data. The pandas read_csv function offers a powerful increase in performance and functionality over this module.

There's more...

The head method accepts a single parameter, n, which controls the number of rows displayed. Similarly, the tail method returns the last n rows.

See also

Previous Section
End of Section 3
Next Section