Book Image

Learning pandas - Second Edition

By : Michael Heydt
Book Image

Learning pandas - Second Edition

By: Michael Heydt

Overview of this book

You will learn how to use pandas to perform data analysis in Python. You will start with an overview of data analysis and iteratively progress from modeling data, to accessing data from remote sources, performing numeric and statistical analysis, through indexing and performing aggregate analysis, and finally to visualizing statistical data and applying pandas to finance. With the knowledge you gain from this book, you will quickly learn pandas and how it can empower you in the exciting world of data manipulation, analysis and science.

Related Content you might be interested in

Current Title:

Small book image
Learning pandas - Second Edition
Michael Heydt
Jun, 2017 | 446 pages
Small book image
SciPy Recipes
Luiz Felipe Martins | V Kishore Ayyadevara | Ruben Oliva Ramos
Dec, 2017 | 386 pages
Small book image
Hands-On Data Analysis with NumPy and Pandas
Curtis Miller
Jun, 2018 | 168 pages
Small book image
Mastering pandas.
Ashish Kumar
Oct, 2019 | 674 pages
Table of Contents (16 chapters)
Preface
Preface
What this book covers
What you need for this book
Who this book is for
Conventions
Reader feedback
Customer support
Free Chapter
1
pandas and Data Analysis
pandas and Data Analysis
Introducing pandas
Data manipulation, analysis, science, and pandas
The process of data analysis
Relating the book to the process
Concepts of data and analysis in our tour of pandas
Other Python libraries of value with pandas
Summary
2
Up and Running with pandas
Up and Running with pandas
Installation of Anaconda
IPython and Jupyter Notebook
Introducing the pandas Series and DataFrame
Visualization
Summary
3
Representing Univariate Data with the Series
Representing Univariate Data with the Series
Configuring pandas
Creating a Series
The .index and .values properties
The size and shape of a Series
Specifying an index at creation
Heads, tails, and takes
Retrieving values in a Series by label or position
Slicing a Series into subsets
Alignment via index labels
Performing Boolean selection
Re-indexing a Series
Modifying a Series in-place
Summary
4
Representing Tabular and Multivariate Data with the DataFrame
Representing Tabular and Multivariate Data with the DataFrame
Configuring pandas
Creating DataFrame objects
Accessing data within a DataFrame
Selecting rows using Boolean selection
Selecting across both rows and columns
Summary
5
Manipulating DataFrame Structure
Manipulating DataFrame Structure
Configuring pandas
Renaming columns
Adding new columns with [] and .insert()
Adding columns through enlargement
Adding columns using concatenation
Reordering columns
Replacing the contents of a column
Deleting columns
Appending new rows
Concatenating rows
Adding and replacing rows via enlargement
Removing rows using .drop()
Removing rows using Boolean selection
Removing rows using a slice
Summary
6
Indexing Data
Indexing Data
Configuring pandas
The importance of indexes
The pandas index types
Working with Indexes
Hierarchical indexing
Summary
7
Categorical Data
Categorical Data
Configuring pandas
Creating Categoricals
Renaming categories
Appending new categories
Removing categories
Removing unused categories
Setting categories
Descriptive information of a Categorical
Munging school grades
Summary
8
Numerical and Statistical Methods
Numerical and Statistical Methods
Configuring pandas
Performing numerical methods on pandas objects
Performing statistical processes on pandas objects
Summary
9
Accessing Data
Accessing Data
Configuring pandas
Working with CSV and text/tabular format data
Reading and writing data in Excel format
Reading and writing JSON files
Reading HTML data from the web
Reading and writing HDF5 format files
Accessing CSV data on the web
Reading and writing from/to SQL databases
Reading data from remote data services
Summary
10
Tidying Up Your Data
Tidying Up Your Data
Configuring pandas
What is tidying your data?
How to work with missing data
Handling duplicate data
Transforming data
Summary
11
Combining, Relating, and Reshaping Data
Combining, Relating, and Reshaping Data
Configuring pandas
Concatenating data in multiple objects
Merging and joining data
Pivoting data to and from value and indexes
Stacking and unstacking
Performance benefits of stacked data
Summary
12
Data Aggregation
Data Aggregation
Configuring pandas
The split, apply, and combine (SAC) pattern
Data for the examples
Splitting data
Applying aggregate functions, transforms, and filters
Transforming groups of data
Filtering groups from aggregation
Summary
13
Time-Series Modelling
Time-Series Modelling
Setting up the IPython notebook
Representation of dates, time, and intervals
Introducing time-series data
Calculating new dates using offsets
Representing durations of time using Period
Handling holidays using calendars
Normalizing timestamps using time zones
Manipulating time-series data
Time-series moving-window operations
Summary
14
Visualization
Visualization
Configuring pandas
Plotting basics with pandas
Creating time-series charts
Common plots used in statistical analyses
Manually rendering multiple plots in a single chart
Summary
15
Historical Stock Price Analysis
Historical Stock Price Analysis
Setting up the IPython notebook
Obtaining and organizing stock data from Google
Plotting time-series prices
Plotting volume-series data
Calculating the simple daily percentage change in closing price
Calculating simple daily cumulative returns of a stock
Resampling data from daily to monthly returns
Analyzing distribution of returns
Performing a moving-average calculation
Comparison of average daily returns across stocks
Correlation of stocks based on the daily percentage change of the closing price
Calculating the volatility of stocks
Determining risk relative to expected returns
Summary

Correlation of stocks based on the daily percentage change of the closing price

Correlation is a measure of the strength of the association between two variables. A correlation coefficient of 1.0 means that every change in value in one set of data has a proportionate change in value in the other set of data. A 0.0 correlation means that the data sets have no relationship. The higher the correlation, the more ability there is to predict a change in each, based on one or the other.

The scatter plot matrix gave us a quick visual idea of the correlation between two stocks, but it was not an exact number. The exact correlation between the columns of data in DataFrame can be calculated using the .corr() method. This will produce a matrix of all possible correlations between the variables represented the columns.

The following example calculates the correlation in the daily percentage...

Previous Section
End of Section 12
Next Section