7. Tidying Up Your Data | Learning Pandas

Sign In Start Free Trial

Book Overview & Buying
Table Of Contents

Learning Pandas

By : Michael Heydt

4.2 (10)

Learning Pandas

4.2 (10)

By: Michael Heydt

Overview of this book

If you are a Python programmer who wants to get started with performing data analysis using pandas and Python, this is the book for you. Some experience with statistical analysis would be helpful but is not mandatory.

Preface

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Free Chapter

1. A Tour of pandas

1. A Tour of pandas

pandas and why it is important

pandas and IPython Notebooks

Referencing pandas in the application

Primary pandas objects

Loading data from files and the Web

Simplicity of visualization of pandas data

Summary

2. Installing pandas

2. Installing pandas

Getting Anaconda

Installing Anaconda

Ensuring pandas is up to date

Running a small pandas sample in IPython

Starting the IPython Notebook server

Installing and running IPython Notebooks

Using Wakari for pandas

Summary

3. NumPy for pandas

3. NumPy for pandas

Installing and importing NumPy

Benefits and characteristics of NumPy arrays

Creating NumPy arrays and performing basic array operations

Selecting array elements

Logical operations on arrays

Slicing arrays

Reshaping arrays

Combining arrays

Splitting arrays

Useful numerical methods of NumPy arrays

Summary

4. The pandas Series Object

4. The pandas Series Object

The Series object

Importing pandas

Creating Series

Size, shape, uniqueness, and counts of values

Peeking at data with heads, tails, and take

Looking up values in Series

Arithmetic operations

The special case of Not-A-Number (NaN)

Boolean selection

Reindexing a Series

Slicing a Series

Summary

5. The pandas DataFrame Object

5. The pandas DataFrame Object

Creating DataFrame from scratch

Example data

Selecting columns of a DataFrame

Selecting rows and values of a DataFrame using the index

Selecting rows of a DataFrame by Boolean selection

Modifying the structure and content of DataFrame

Arithmetic on a DataFrame

Resetting and reindexing

Hierarchical indexing

Summarized data and descriptive statistics

Summary

6. Accessing Data

6. Accessing Data

Setting up the IPython notebook

Reading and writing JSON files

Accessing data on the web and in the cloud

Reading and writing from/to SQL databases

Reading data from remote data services

Summary

7. Tidying Up Your Data

7. Tidying Up Your Data

What is tidying your data?

Setting up the IPython notebook

Working with missing data

Handling duplicate data

Transforming Data

Summary

8. Combining and Reshaping Data

8. Combining and Reshaping Data

Setting up the IPython notebook

Concatenating data

Merging and joining data

Stacking and unstacking

Performance benefits of stacked data

Summary

9. Grouping and Aggregating Data

9. Grouping and Aggregating Data

Setting up the IPython notebook

The split, apply, and combine (SAC) pattern

Split

Apply

Discretization and Binning

Summary

10. Time-series Data

10. Time-series Data

Setting up the IPython notebook

Representation of dates, time, and intervals

Introducing time-series data

Calculating new dates using offsets

Handling holidays using calendars

Normalizing timestamps using time zones

Manipulating time-series data

Summary

11. Visualization

11. Visualization

Setting up the IPython notebook

Plotting basics with pandas

Common plots used in statistical analyses

Multiple plots in a single chart

Summary

12. Applications to Finance

12. Applications to Finance

Setting up the IPython notebook

Obtaining and organizing stock data from Yahoo!

Plotting time-series prices

Performing a moving-average calculation

Volatility calculation

Determining risk relative to expected returns

Summary

Index

Index

Working with missing data

Data is "missing" in pandas when it has a value of NaN (also seen as np.nan—the form from NumPy). The NaN value represents that in a particular Series that there is not a value specified for the particular index label.

In pandas, there are a number of reasons why a value can be NaN:

A join of two sets of data does not have matched values
Data that you retrieved from an external source is incomplete
The NaN value is not known at a given point in time and will be filled in later
There is a data collection error retrieving a value, but the event must still be recorded in the index
Reindexing of data has resulted in an index that does not have a value
The shape of data has changed and there are now additional rows or columns, which at the time of reshaping could not be determined

There are likely more reasons, but the general point is that they occur and you, as a pandas programmer, will need to work with them effectively to be able to perform correct data analysis. Fortunately...

CONTINUE READING

83

Tech Concepts

36

Programming languages

73

Tech Tools

Unlimited access to the largest independent learning library in tech of over 8,000 expert-authored tech books and videos.

Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.

50+ new titles added per month and exclusive early access to books as they are being written.

Learning Pandas

Search

Your notes and bookmarks