Book Image

Pandas Cookbook

By : Theodore Petrou
Book Image

Pandas Cookbook

By: Theodore Petrou

Overview of this book

This book will provide you with unique, idiomatic, and fun recipes for both fundamental and advanced data manipulation tasks with pandas 0.20. Some recipes focus on achieving a deeper understanding of basic principles, or comparing and contrasting two similar operations. Other recipes will dive deep into a particular dataset, uncovering new and unexpected insights along the way. The pandas library is massive, and it's common for frequent users to be unaware of many of its more impressive features. The official pandas documentation, while thorough, does not contain many useful examples of how to piece together multiple commands like one would do during an actual analysis. This book guides you, as if you were looking over the shoulder of an expert, through practical situations that you are highly likely to encounter. Many advanced recipes combine several different features across the pandas 0.20 library to generate results.

Related Content you might be interested in

Current Title:

Small book image
Pandas Cookbook
Theodore Petrou
Oct, 2017 | 532 pages
Small book image
Learning pandas
Michael Heydt
Jun, 2017 | 446 pages
Small book image
Mastering Exploratory Analysis with pandas
Harish Garg
Sep, 2018 | 140 pages
Small book image
Python Data Cleaning Cookbook
Michael Walker
Dec, 2020 | 436 pages
Table of Contents (12 chapters)
Preface
Preface
What this book covers
What you need for this book
Who this book is for
Conventions
Assumptions for every recipe
Dataset Descriptions
Sections
Reader feedback
Customer support
Free Chapter
1
Pandas Foundations
Pandas Foundations
Introduction
Dissecting the anatomy of a DataFrame
Accessing the main DataFrame components
Understanding data types
Selecting a single column of data as a Series
Calling Series methods
Working with operators on a Series
Chaining Series methods together
Making the index meaningful
Renaming row and column names
Creating and deleting columns
2
Essential DataFrame Operations
Essential DataFrame Operations
Introduction
Selecting multiple DataFrame columns
Selecting columns with methods
Ordering column names sensibly
Operating on the entire DataFrame
Chaining DataFrame methods together
Working with operators on a DataFrame
Comparing missing values
Transposing the direction of a DataFrame operation
Determining college campus diversity
3
Beginning Data Analysis
Beginning Data Analysis
Introduction
Developing a data analysis routine
Reducing memory by changing data types
Selecting the smallest of the largest
Selecting the largest of each group by sorting
Replicating nlargest with sort_values
Calculating a trailing stop order price
4
Selecting Subsets of Data
Selecting Subsets of Data
Introduction
Selecting Series data
Selecting DataFrame rows
Selecting DataFrame rows and columns simultaneously
Selecting data with both integers and labels
Speeding up scalar selection
Slicing rows lazily
Slicing lexicographically
5
Boolean Indexing
Boolean Indexing
Introduction
Calculating boolean statistics
Constructing multiple boolean conditions
Filtering with boolean indexing
Replicating boolean indexing with index selection
Selecting with unique and sorted indexes
Gaining perspective on stock prices
Translating SQL WHERE clauses
Determining the normality of stock market returns
Improving readability of boolean indexing with the query method
Preserving Series with the where method
Masking DataFrame rows
Selecting with booleans, integer location, and labels
6
Index Alignment
Index Alignment
Introduction
Examining the Index object
Producing Cartesian products
Exploding indexes
Filling values with unequal indexes
Appending columns from different DataFrames
Highlighting the maximum value from each column
Replicating idxmax with method chaining
Finding the most common maximum
7
Grouping for Aggregation, Filtration, and Transformation
Grouping for Aggregation, Filtration, and Transformation
Introduction
Defining an aggregation
Grouping and aggregating with multiple columns and functions
Removing the MultiIndex after grouping
Customizing an aggregation function
Customizing aggregating functions with *args and **kwargs
Examining the groupby object
Filtering for states with a minority majority
Transforming through a weight loss bet
Calculating weighted mean SAT scores per state with apply
Grouping by continuous variables
Counting the total number of flights between cities
Finding the longest streak of on-time flights
8
Restructuring Data into a Tidy Form
Restructuring Data into a Tidy Form
Introduction
Tidying variable values as column names with stack
Tidying variable values as column names with melt
Stacking multiple groups of variables simultaneously
Inverting stacked data
Unstacking after a groupby aggregation
Replicating pivot_table with a groupby aggregation
Renaming axis levels for easy reshaping
Tidying when multiple variables are stored as column names
Tidying when multiple variables are stored as column values
Tidying when two or more values are stored in the same cell
Tidying when variables are stored in column names and values
Tidying when multiple observational units are stored in the same table
9
Combining Pandas Objects
Combining Pandas Objects
Introduction
Appending new rows to DataFrames
Concatenating multiple DataFrames together
Comparing President Trump's and Obama's approval ratings
Understanding the differences between concat, join, and merge
Connecting to SQL databases
10
Time Series Analysis
Time Series Analysis
Introduction
Understanding the difference between Python and pandas date tools
Slicing time series intelligently
Using methods that only work with a DatetimeIndex
Counting the number of weekly crimes
Aggregating weekly crime and traffic accidents separately
Measuring crime by weekday and year
Grouping with anonymous functions with a DatetimeIndex
Grouping by a Timestamp and another column
Finding the last time crime was 20% lower with merge_asof
11
Visualization with Matplotlib, Pandas, and Seaborn
Visualization with Matplotlib, Pandas, and Seaborn
Introduction
Getting started with matplotlib
Visualizing data with matplotlib
Plotting basics with pandas
Visualizing the flights dataset
Stacking area charts to discover emerging trends
Understanding the differences between seaborn and pandas
Doing multivariate analysis with seaborn Grids
Uncovering Simpson's paradox in the diamonds dataset with seaborn

What you need for this book

Pandas is a third-party package for the Python programming language and, as of the printing of this book, is on version 0.20. Currently, Python has two major supported releases, versions 2.7 and 3.6. Python 3 is the future, and it is now highly recommended that all scientific computing users of Python use it, as Python 2 will no longer be supported in 2020. All examples in this book have been run and tested with pandas 0.20 on Python 3.6.

In addition to pandas, you will need to have the matplotlib version 2.0 and seaborn version 0.8 visualization libraries installed. A major dependence for pandas is the NumPy library, which forms the basis of most of the popular Python scientific computing libraries.

There are a wide variety of ways in which you can install pandas and the rest of the libraries mentioned on your computer, but by far the simplest method is to install the Anaconda distribution. Created by Continuum Analytics, it packages together all the popular libraries for scientific computing in a single downloadable file available on Windows, Mac OSX, and Linux. Visit the download page to get the Anaconda distribution (https://www.anaconda.com/download).

In addition to all the scientific computing libraries, the Anaconda distribution comes with Jupyter Notebook, which is a browser-based program for developing in Python, among many other languages. All of the recipes for this book were developed inside of a Jupyter Notebook and all of the individual notebooks for each chapter will be available for you to use.

It is possible to install all the necessary libraries for this book without the use of the Anaconda distribution. For those that are interested, visit the pandas Installation page (http://pandas.pydata.org/pandas-docs/stable/install.html).

Running a Jupyter Notebook

The suggested method to work through the content of this book is to have a Jupyter Notebook up and running so that you can run the code while reading through the recipes. This allows you to go exploring on your own and gain a deeper understanding than by just reading the book alone.

Assuming that you have installed the Anaconda distribution on your machine, you have two options available to start the Jupyter Notebook:

  • Use the program Anaconda Navigator
  • Run the jupyter notebook command from the Terminal/Command Prompt

The Anaconda Navigator is a GUI-based tool that allows you to find all the different software provided by Anaconda with ease. Running the program will give you a screen like this:

As you can see, there are many programs available to you. Click Launch to open the Jupyter Notebook. A new tab will open in your browser, showing you a list of folders and files in your home directory:

Instead of using the Anaconda Navigator, you can launch Jupyter Notebook by opening up your Terminal/Command Prompt and running the jupyter notebook command like this:

It is not necessary to run this command from your home directory. You can run it from any location, and the contents in the browser will reflect that location.

Although we have now started the Jupyter Notebook program, we haven't actually launched a single individual notebook where we can start developing in Python. To do so, you can click on the New button on the right-hand side of the page, which will drop down a list of all the possible kernels available for you to use. If you just downloaded Anaconda, then you will only have a single kernel available to you (Python 3). After selecting the Python 3 kernel, a new tab will open in the browser, where you can start writing Python code:

You can, of course, open previously created notebooks instead of beginning a new one. To do so, simply navigate through the filesystem provided in the Jupyter Notebook browser home page and select the notebook you want to open. All Jupyter Notebook files end in .ipynb. For instance, when you navigate to the location of the notebook files for this book, you will see all of them like this:

Previous Section
Next Section