Book Image

Hands-On Data Analysis with Pandas

By : Stefanie Molin
Book Image

Hands-On Data Analysis with Pandas

By: Stefanie Molin

Overview of this book

Data analysis has become a necessary skill in a variety of domains where knowing how to work with data and extract insights can generate significant value. Hands-On Data Analysis with Pandas will show you how to analyze your data, get started with machine learning, and work effectively with Python libraries often used for data science, such as pandas, NumPy, matplotlib, seaborn, and scikit-learn. Using real-world datasets, you will learn how to use the powerful pandas library to perform data wrangling to reshape, clean, and aggregate your data. Then, you will be able to conduct exploratory data analysis by calculating summary statistics and visualizing the data to find patterns. In the concluding chapters, you will explore some applications of anomaly detection, regression, clustering, and classification using scikit-learn to make predictions based on past data. By the end of this book, you will be equipped with the skills you need to use pandas to ensure the veracity of your data, visualize it for effective decision-making, and reliably reproduce analyses across multiple datasets.

Related Content you might be interested in

Current Title:

Small book image
Hands-On Data Analysis with Pandas
Stefanie Molin
Jul, 2019 | 740 pages
Small book image
Hands-On Exploratory Data Analysis with Python
Usman Ahmed | Suresh Kumar Mukhiya
Mar, 2020 | 352 pages
Small book image
Pandas Cookbook
Theodore Petrou
Oct, 2017 | 532 pages
Small book image
Learning pandas
Michael Heydt
Jun, 2017 | 446 pages
Table of Contents (21 chapters)
Preface
Preface
Who this book is for
What this book covers
To get the most out of this book
Conventions used
Get in touch
Free Chapter
1
Section 1: Getting Started with Pandas
Section 1: Getting Started with Pandas
2
Introduction to Data Analysis
Introduction to Data Analysis
Chapter materials
Fundamentals of data analysis
Statistical foundations
Setting up a virtual environment
Summary
Exercises
Further reading
3
Working with Pandas DataFrames
Working with Pandas DataFrames
Chapter materials
Pandas data structures
Bringing data into a pandas DataFrame
Inspecting a DataFrame object
Grabbing subsets of the data
Adding and removing data
Summary
Exercises
Further reading
4
Section 2: Using Pandas for Data Analysis
Section 2: Using Pandas for Data Analysis
5
Data Wrangling with Pandas
Data Wrangling with Pandas
Chapter materials
What is data wrangling?
Collecting temperature data
Cleaning up the data
Restructuring the data
Handling duplicate, missing, or invalid data
Summary
Exercises
Further reading
6
Aggregating Pandas DataFrames
Aggregating Pandas DataFrames
Chapter materials
Database-style operations on DataFrames
DataFrame operations
Aggregations with pandas and numpy
Time series
Summary
Exercises
Further reading
7
Visualizing Data with Pandas and Matplotlib
Visualizing Data with Pandas and Matplotlib
Chapter materials
An introduction to matplotlib
Plotting with pandas
The pandas.plotting subpackage
Summary
Exercises
Further reading
8
Plotting with Seaborn and Customization Techniques
Plotting with Seaborn and Customization Techniques
Chapter materials
Utilizing seaborn for advanced plotting
Formatting
Customizing visualizations
Summary
Exercises
Further reading
9
Section 3: Applications - Real-World Analyses Using Pandas
Section 3: Applications - Real-World Analyses Using Pandas
10
Financial Analysis - Bitcoin and the Stock Market
Financial Analysis - Bitcoin and the Stock Market
Chapter materials
Building a Python package
Data extraction with pandas
Exploratory data analysis
Technical analysis of financial instruments
Modeling performance
Summary
Exercises
Further reading
11
Rule-Based Anomaly Detection
Rule-Based Anomaly Detection
Chapter materials
Simulating login attempts
Exploratory data analysis
Rule-based anomaly detection
Summary
Exercises
Further reading
12
Section 4: Introduction to Machine Learning with Scikit-Learn
Section 4: Introduction to Machine Learning with Scikit-Learn
13
Getting Started with Machine Learning in Python
Getting Started with Machine Learning in Python
Chapter materials
Learning the lingo
Exploratory data analysis
Preprocessing data
Clustering
Regression
Classification
Summary
Exercises
Further reading
14
Making Better Predictions - Optimizing Models
Making Better Predictions - Optimizing Models
Chapter materials
Hyperparameter tuning with grid search
Feature engineering
Ensemble methods
Inspecting classification prediction confidence
Addressing class imbalance
Regularization
Summary
Exercises
Further reading
15
Machine Learning Anomaly Detection
Machine Learning Anomaly Detection
Chapter materials
Exploring the data
Unsupervised methods
Supervised methods
Online learning
Summary
Exercises
Further reading
16
Section 5: Additional Resources
Section 5: Additional Resources
17
The Road Ahead
The Road Ahead
Data resources
Practicing working with data
Python practice
Summary
Exercises
Further reading
18
Solutions
Solutions
19
Other Books You May Enjoy
Other Books You May Enjoy
Leave a review - let other readers know what you think
Appendix
Appendix
Data analysis workflow
Choosing the appropriate visualization
Machine learning workflow

Exercises

Complete the following exercises for some practice with the machine learning workflow and exposure to some additional anomaly detection strategies:

  1. A one-class SVM is another model that can be used for unsupervised outlier detection. Build a one-class SVM model with the default parameters, using a pipeline with the StandardScaler and the OneClassSVM. Train the model on January 2018 data, just as we did for the isolation forest. Make predictions on that same data. Count the number of inliers and outliers this model identifies.
  2. Using the 2018 minutely data, build a k-means model with two clusters after standardizing the data with the StandardScaler. With the labeled data in the attacks table in the SQLite database (logs/logs.db), see whether this model gets a good Fowlkes-Mallows score (use the fowlkes_mallows_score() function in sklearn.metrics).
  3. Evaluate the performance...
Previous Section
End of Section 8
Next Section