Sign In Start Free Trial

Book Overview & Buying
Table Of Contents

Pandas Cookbook - Third Edition

By : William Ayd, Matthew Harrison

4.9 (11)

Pandas Cookbook

4.9 (11)

By: William Ayd, Matthew Harrison

Overview of this book

Unlock the full power of pandas 2.x with this hands-on cookbook, designed for Python developers, data analysts, and data scientists who need fast, efficient solutions for real-world data challenges. This book provides practical, ready-to-use recipes to streamline your workflow. With step-by-step guidance, you'll master data wrangling, visualization, performance optimization, and scalable data analysis using pandas’ most powerful features. From importing and merging large datasets to advanced time series analysis and SQL-like operations, this cookbook equips you with the tools to analyze, manipulate, and visualize data like a pro. Learn how to boost efficiency, optimize memory usage, and seamlessly integrate pandas with NumPy, PyArrow, and databases. This book will help you transform raw data into actionable insights with ease. *Email sign-up and proof of purchase required

Preface

Preface

Who this book is for

What this book covers

To get the most out of this book

Sections

Get in touch

Reviews

Leave a Review!

Making the Most Out of This Book – Get to Know Your Free Benefits

Free Chapter

pandas Foundations

pandas Foundations

Importing pandas

Series

DataFrame

Index

Series attributes

DataFrame attributes

Selection and Assignment

Selection and Assignment

Basic selection from a Series

Basic selection from a DataFrame

Position-based selection of a Series

Position-based selection of a DataFrame

Label-based selection from a Series

Label-based selection from a DataFrame

Mixing position-based and label-based selection

DataFrame.filter

Selection by data type

Selection/filtering via Boolean arrays

Selection with a MultiIndex – A single level

Selection with a MultiIndex – Multiple levels

Selection with a MultiIndex – a DataFrame

Item assignment with .loc and .iloc

DataFrame column assignment

Data Types

Data Types

Integral types

Floating point types

Boolean types

String types

Missing value handling

Categorical types

Temporal types – datetime

Temporal types – timedelta

Temporal PyArrow types

PyArrow List types

PyArrow decimal types

NumPy type system, the object type, and pitfalls

The pandas I/O System

The pandas I/O System

CSV – basic reading/writing

CSV – strategies for reading large files

Microsoft Excel – basic reading/writing

Microsoft Excel – finding tables in non-default locations

Microsoft Excel – hierarchical data

SQL using SQLAlchemy

SQL using ADBC

Apache Parquet

JSON

HTML

Pickle

Third-party I/O libraries

Algorithms and How to Apply Them

Algorithms and How to Apply Them

Basic pd.Series arithmetic

Basic pd.DataFrame arithmetic

Aggregations

Transformations

Map

Apply

Summary statistics

Binning algorithms

One-hot encoding with pd.get_dummies

Chaining with .pipe

Selecting the lowest-budget movies from the top 100

Calculating a trailing stop order price

Finding the baseball players best at…

Understanding which position scores the most per tea

Visualization

Visualization

Creating charts from aggregated data

Plotting distributions of non-aggregated data

Further plot customization with Matplotlib

Exploring scatter plots

Exploring categorical data

Exploring continuous data

Using seaborn for advanced plots

Reshaping DataFrames

Reshaping DataFrames

Concatenating pd.DataFrame objects

Merging DataFrames with pd.merge

Joining DataFrames with pd.DataFrame.join

Reshaping with pd.DataFrame.stack and pd.DataFrame.unstack

Reshaping with pd.DataFrame.melt

Reshaping with pd.wide_to_long

Reshaping with pd.DataFrame.pivot and pd.pivot_table

Reshaping with pd.DataFrame.explode

Transposing with pd.DataFrame.T

Group By

Group By

Group by basics

Grouping and calculating multiple columns

Group by apply

Window operations

Selecting the highest rated movies by year

Comparing the best hitter in baseball across years

Temporal Data Types and Algorithms

Temporal Data Types and Algorithms

Timezone handling

DateOffsets

Datetime selection

Resampling

Aggregating weekly crime and traffic accidents

Calculating year-over-year changes in crime by category

Accurately measuring sensor-collected events with missing values

General Usage and Performance Tips

General Usage and Performance Tips

Avoid dtype=object

Be cognizant of data sizes

Use vectorized functions instead of loops

Avoid mutating data

Dictionary-encode low cardinality data

Test-driven development features

The pandas Ecosystem

The pandas Ecosystem

Foundational libraries

Exploratory data analysis

Data validation

Visualization

Data science

Databases

Other DataFrame libraries

Other Books You May Enjoy

Other Books You May Enjoy

Index

Index

Temporal PyArrow types

At this point, we have reviewed many of the “first-class” data types built into pandas, while highlighting some rough edges and inconsistencies that plague them. Despite those issues, the types baked into pandas can take you a long way in your data journey.

But there are still cases where the pandas types are not suitable, with a common case being interoperability with databases. Most databases have distinct DATE and DATETIME types, so the fact that pandas only offers a DATETIME type can be disappointing to users fluent in SQL.

Fortunately, the Apache Arrow project defines a true DATE type. Starting in version 2.0, pandas users can start leveraging Arrow types exposed through the PyArrow library.

How to do it

To construct PyArrow types in pandas directly, you will always provide a dtype= argument of the pd.ArrowDtype(XXX) form, replacing XXX with the appropriate PyArrow type. The DATE type in PyArrow is called pa.date32():

ser...

CONTINUE READING

83

Tech Concepts

36

Programming languages

73

Tech Tools

Unlimited access to the largest independent learning library in tech of over 8,000 expert-authored tech books and videos.

Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.

50+ new titles added per month and exclusive early access to books as they are being written.

Pandas Cookbook

Search

Your notes and bookmarks