Book Image

Learning pandas - Second Edition

By : Michael Heydt
Book Image

Learning pandas - Second Edition

By: Michael Heydt

Overview of this book

You will learn how to use pandas to perform data analysis in Python. You will start with an overview of data analysis and iteratively progress from modeling data, to accessing data from remote sources, performing numeric and statistical analysis, through indexing and performing aggregate analysis, and finally to visualizing statistical data and applying pandas to finance. With the knowledge you gain from this book, you will quickly learn pandas and how it can empower you in the exciting world of data manipulation, analysis and science.

Related Content you might be interested in

Current Title:

Small book image
Learning pandas - Second Edition
Michael Heydt
Jun, 2017 | 446 pages
Small book image
SciPy Recipes
Luiz Felipe Martins | V Kishore Ayyadevara | Ruben Oliva Ramos
Dec, 2017 | 386 pages
Small book image
Hands-On Data Analysis with NumPy and Pandas
Curtis Miller
Jun, 2018 | 168 pages
Small book image
Mastering pandas.
Ashish Kumar
Oct, 2019 | 674 pages
Table of Contents (16 chapters)
Preface
Preface
What this book covers
What you need for this book
Who this book is for
Conventions
Reader feedback
Customer support
Free Chapter
1
pandas and Data Analysis
pandas and Data Analysis
Introducing pandas
Data manipulation, analysis, science, and pandas
The process of data analysis
Relating the book to the process
Concepts of data and analysis in our tour of pandas
Other Python libraries of value with pandas
Summary
2
Up and Running with pandas
Up and Running with pandas
Installation of Anaconda
IPython and Jupyter Notebook
Introducing the pandas Series and DataFrame
Visualization
Summary
3
Representing Univariate Data with the Series
Representing Univariate Data with the Series
Configuring pandas
Creating a Series
The .index and .values properties
The size and shape of a Series
Specifying an index at creation
Heads, tails, and takes
Retrieving values in a Series by label or position
Slicing a Series into subsets
Alignment via index labels
Performing Boolean selection
Re-indexing a Series
Modifying a Series in-place
Summary
4
Representing Tabular and Multivariate Data with the DataFrame
Representing Tabular and Multivariate Data with the DataFrame
Configuring pandas
Creating DataFrame objects
Accessing data within a DataFrame
Selecting rows using Boolean selection
Selecting across both rows and columns
Summary
5
Manipulating DataFrame Structure
Manipulating DataFrame Structure
Configuring pandas
Renaming columns
Adding new columns with [] and .insert()
Adding columns through enlargement
Adding columns using concatenation
Reordering columns
Replacing the contents of a column
Deleting columns
Appending new rows
Concatenating rows
Adding and replacing rows via enlargement
Removing rows using .drop()
Removing rows using Boolean selection
Removing rows using a slice
Summary
6
Indexing Data
Indexing Data
Configuring pandas
The importance of indexes
The pandas index types
Working with Indexes
Hierarchical indexing
Summary
7
Categorical Data
Categorical Data
Configuring pandas
Creating Categoricals
Renaming categories
Appending new categories
Removing categories
Removing unused categories
Setting categories
Descriptive information of a Categorical
Munging school grades
Summary
8
Numerical and Statistical Methods
Numerical and Statistical Methods
Configuring pandas
Performing numerical methods on pandas objects
Performing statistical processes on pandas objects
Summary
9
Accessing Data
Accessing Data
Configuring pandas
Working with CSV and text/tabular format data
Reading and writing data in Excel format
Reading and writing JSON files
Reading HTML data from the web
Reading and writing HDF5 format files
Accessing CSV data on the web
Reading and writing from/to SQL databases
Reading data from remote data services
Summary
10
Tidying Up Your Data
Tidying Up Your Data
Configuring pandas
What is tidying your data?
How to work with missing data
Handling duplicate data
Transforming data
Summary
11
Combining, Relating, and Reshaping Data
Combining, Relating, and Reshaping Data
Configuring pandas
Concatenating data in multiple objects
Merging and joining data
Pivoting data to and from value and indexes
Stacking and unstacking
Performance benefits of stacked data
Summary
12
Data Aggregation
Data Aggregation
Configuring pandas
The split, apply, and combine (SAC) pattern
Data for the examples
Splitting data
Applying aggregate functions, transforms, and filters
Transforming groups of data
Filtering groups from aggregation
Summary
13
Time-Series Modelling
Time-Series Modelling
Setting up the IPython notebook
Representation of dates, time, and intervals
Introducing time-series data
Calculating new dates using offsets
Representing durations of time using Period
Handling holidays using calendars
Normalizing timestamps using time zones
Manipulating time-series data
Time-series moving-window operations
Summary
14
Visualization
Visualization
Configuring pandas
Plotting basics with pandas
Creating time-series charts
Common plots used in statistical analyses
Manually rendering multiple plots in a single chart
Summary
15
Historical Stock Price Analysis
Historical Stock Price Analysis
Setting up the IPython notebook
Obtaining and organizing stock data from Google
Plotting time-series prices
Plotting volume-series data
Calculating the simple daily percentage change in closing price
Calculating simple daily cumulative returns of a stock
Resampling data from daily to monthly returns
Analyzing distribution of returns
Performing a moving-average calculation
Comparison of average daily returns across stocks
Correlation of stocks based on the daily percentage change of the closing price
Calculating the volatility of stocks
Determining risk relative to expected returns
Summary

Appending new rows

Appending of rows is performed using the .append() method of the DataFrame. The process of appending returns a new DataFrame with the data from the original DataFrame added first and then rows from the second. Appending does not perform alignment and can result in duplicate index labels.

The following code demonstrates appending two DataFrame objects extracted from the sp500 data. The first DataFrame consists of rows (by position) 0, 1 and 2, and the second consists of rows (also by position) 10, 11 and 2. The row at position 2 (with label ABBV) is included in both to demonstrate the creation of duplicate index labels.

The set of columns of the DataFrame objects used in an append do not need to be the same. The resulting data frame will consist of the union of the columns in both, with missing column data filled with NaN. The following demonstrates this by...

Previous Section
End of Section 10
Next Section