Pandas 1.x Cookbook - Second Edition

By : Matt Harrison, Theodore Petrou

Pandas 1.x Cookbook - Second Edition

By: Matt Harrison, Theodore Petrou

Overview of this book

The pandas library is massive, and it's common for frequent users to be unaware of many of its more impressive features. The official pandas documentation, while thorough, does not contain many useful examples of how to piece together multiple commands as one would do during an actual analysis. This book guides you, as if you were looking over the shoulder of an expert, through situations that you are highly likely to encounter. This new updated and revised edition provides you with unique, idiomatic, and fun recipes for both fundamental and advanced data manipulation tasks with pandas. Some recipes focus on achieving a deeper understanding of basic principles, or comparing and contrasting two similar operations. Other recipes will dive deep into a particular dataset, uncovering new and unexpected insights along the way. Many advanced recipes combine several different features across the pandas library to generate results.

Preface

Who this book is for

What this book covers

To get the most out of this book

Pandas Foundations

Understanding data types

Selecting a column

Calling Series methods

Series operations

Chaining Series methods

Renaming column names

Creating and deleting columns

Free Chapter

Essential DataFrame Operations

Introduction

Selecting multiple DataFrame columns

Selecting columns with methods

Ordering column names

Summarizing a DataFrame

Chaining DataFrame methods

DataFrame operations

Comparing missing values

Transposing the direction of a DataFrame operation

Determining college campus diversity

Creating and Persisting DataFrames

Introduction

Creating DataFrames from scratch

Writing CSV

Reading large CSV files

Using Excel files

Working with ZIP files

Working with databases

Reading JSON

Reading HTML tables

Beginning Data Analysis

Introduction

Developing a data analysis routine

Data dictionaries

Reducing memory by changing data types

Selecting the smallest of the largest

Selecting the largest of each group by sorting

Replicating nlargest with sort_values

Calculating a trailing stop order price

Exploratory Data Analysis

Comparing continuous values across categories

Comparing two continuous columns

Comparing categorical values with categorical values

Using the pandas profiling library

Selecting Subsets of Data

Introduction

Selecting Series data

Selecting DataFrame rows

Selecting DataFrame rows and columns simultaneously

Selecting data with both integers and labels

Slicing lexicographically

Filtering Rows

Introduction

Calculating Boolean statistics

Constructing multiple Boolean conditions

Filtering with Boolean arrays

Comparing row filtering and index filtering

Selecting with unique and sorted indexes

Translating SQL WHERE clauses

Improving the readability of Boolean indexing with the query method

Preserving Series size with the .where method

Masking DataFrame rows

Selecting with Booleans, integer location, and labels

Index Alignment

Introduction

Examining the Index object

Producing Cartesian products

Exploding indexes

Filling values with unequal indexes

Adding columns from different DataFrames

Highlighting the maximum value from each column

Replicating idxmax with method chaining

Finding the most common maximum of columns

Grouping for Aggregation, Filtration, and Transformation

Introduction

Defining an aggregation

Grouping and aggregating with multiple columns and functions

Removing the MultiIndex after grouping

Grouping with a custom aggregation function

Customizing aggregating functions with *args and **kwargs

Examining the groupby object

Filtering for states with a minority majority

Transforming through a weight loss bet

Calculating weighted mean SAT scores per state with apply

Grouping by continuous variables

Counting the total number of flights between cities

Finding the longest streak of on-time flights

Restructuring Data into a Tidy Form

Introduction

Tidying variable values as column names with stack

Tidying variable values as column names with melt

Stacking multiple groups of variables simultaneously

Inverting stacked data

Unstacking after a groupby aggregation

Replicating pivot_table with a groupby aggregation

Renaming axis levels for easy reshaping

Tidying when multiple variables are stored as column names

Tidying when multiple variables are stored as a single column

Tidying when two or more values are stored in the same cell

Tidying when variables are stored in column names and values

Combining Pandas Objects

Introduction

Appending new rows to DataFrames

Concatenating multiple DataFrames together

Understanding the differences between concat, join, and merge

Connecting to SQL databases

Time Series Analysis

Introduction

Understanding the difference between Python and pandas date tools

Slicing time series intelligently

Filtering columns with time data

Using methods that only work with a DatetimeIndex

Counting the number of weekly crimes

Aggregating weekly crime and traffic accidents separately

Measuring crime by weekday and year

Grouping with anonymous functions with a DatetimeIndex

Grouping by a Timestamp and another column

Visualization with Matplotlib, Pandas, and Seaborn

Introduction

Getting started with matplotlib

Object-oriented guide to matplotlib

Visualizing data with matplotlib

Plotting basics with pandas

Visualizing the flights dataset

Stacking area charts to discover emerging trends

Understanding the differences between seaborn and pandas

Multivariate analysis with seaborn Grids

Uncovering Simpson's Paradox in the diamonds dataset with seaborn

Debugging and Testing Pandas

Code to transform data

Apply performance

Improving apply performance with Dask, Pandarell, Swifter, and more

Inspecting code

Debugging in Jupyter

Managing data integrity with Great Expectations

Using pytest with pandas

Generating tests with Hypothesis

Other Books You May Enjoy

Index

Customer Reviews

5 star

4 star

3 star

2 star

1 star

Reading large CSV files

The pandas library is an in-memory tool. You need to be able to fit your data in memory to use pandas with it. If you come across a large CSV file that you want to process, you have a few options. If you can process portions of it at a time, you can read it into chunks and process each chunk. Alternatively, if you know that you should have enough memory to load the file, there are a few hints to help pare down the file size.

Note that in general, you should have three to ten times the amount of memory as the size of the DataFrame that you want to manipulate. Extra memory should give you enough extra space to perform many of the common operations.

How to do it...

In this section, we will look at the diamonds dataset. This dataset easily fits into the memory of my 2015 MacBook, but let's pretend that the file is a lot bigger than it is, or that the memory of my machine is limited such that when pandas tries to load it with the read_csv function...

Pandas 1.x Cookbook - Second Edition

By : Matt Harrison, Theodore Petrou

Pandas 1.x Cookbook - Second Edition

By: Matt Harrison, Theodore Petrou

Overview of this book

Related Content you might be interested in

Current Title:

Pandas 1.x Cookbook - Second Edition

Learning pandas

Mastering Exploratory Analysis with pandas

Python Data Cleaning Cookbook

Reading large CSV files

How to do it...