Practical Data Analysis Cookbook

Practical Data Analysis Cookbook

By : Tomasz Drabas

Buy this Book

Practical Data Analysis Cookbook

By: Tomasz Drabas

Buy this Book

Overview of this book

Data analysis is the process of systematically applying statistical and logical techniques to describe and illustrate, condense and recap, and evaluate data. Its importance has been most visible in the sector of information and communication technologies. It is an employee asset in almost all economy sectors. This book provides a rich set of independent recipes that dive into the world of data analytics and modeling using a variety of approaches, tools, and algorithms. You will learn the basics of data handling and modeling, and will build your skills gradually toward more advanced topics such as simulations, raw text processing, social interactions analysis, and more. First, you will learn some easy-to-follow practical techniques on how to read, write, clean, reformat, explore, and understand your data—arguably the most time-consuming (and the most important) tasks for any data scientist. In the second section, different independent recipes delve into intermediate topics such as classification, clustering, predicting, and more. With the help of these easy-to-follow recipes, you will also learn techniques that can easily be expanded to solve other real-life problems such as building recommendation engines or predictive models. In the third section, you will explore more advanced topics: from the field of graph theory through natural language processing, discrete choice modeling to simulations. You will also get to expand your knowledge on identifying fraud origin with the help of a graph, scrape Internet websites, and classify movies based on their reviews. By the end of this book, you will be able to efficiently use the vast array of tools that the Python environment has to offer.

Practical Data Analysis Cookbook

Credits

About the Author

Acknowledgments

About the Reviewers

www.PacktPub.com

Preface

Free Chapter

Preparing the Data

Introduction

Reading and writing CSV/TSV files with Python

Reading and writing JSON files with Python

Reading and writing Excel files with Python

Reading and writing XML files with Python

Retrieving HTML pages with pandas

Storing and retrieving from a relational database

Storing and retrieving from MongoDB

Opening and transforming data with OpenRefine

Exploring the data with Open Refine

Removing duplicates

Using regular expressions and GREL to clean up data

Imputing missing observations

Normalizing and standardizing the features

Binning the observations

Encoding categorical variables

Exploring the Data

Introduction

Producing descriptive statistics

Exploring correlations between features

Visualizing the interactions between features

Producing histograms

Creating multivariate charts

Sampling the data

Splitting the dataset into training, cross-validation, and testing

Classification Techniques

Introduction

Testing and comparing the models

Classifying with Naïve Bayes

Using logistic regression as a universal classifier

Utilizing Support Vector Machines as a classification engine

Classifying calls with decision trees

Predicting subscribers with random tree forests

Employing neural networks to classify calls

Clustering Techniques

Introduction

Assessing the performance of a clustering method

Clustering data with k-means algorithm

Finding an optimal number of clusters for k-means

Discovering clusters with mean shift clustering model

Building fuzzy clustering model with c-means

Using hierarchical model to cluster your data

Finding groups of potential subscribers with DBSCAN and BIRCH algorithms

Reducing Dimensions

Introduction

Creating three-dimensional scatter plots to present principal components

Reducing the dimensions using the kernel version of PCA

Using Principal Component Analysis to find things that matter

Finding the principal components in your data using randomized PCA

Extracting the useful dimensions using Linear Discriminant Analysis

Using various dimension reduction techniques to classify calls using the k-Nearest Neighbors classification model

Regression Methods

Introduction

Identifying and tackling multicollinearity

Building Linear Regression model

Using OLS to forecast how much electricity can be produced

Estimating the output of an electric plant using CART

Employing the kNN model in a regression problem

Applying the Random Forest model to a regression analysis

Gauging the amount of electricity a plant can produce using SVMs

Training a Neural Network to predict the output of a power plant

Time Series Techniques

Introduction

Handling date objects in Python

Understanding time series data

Smoothing and transforming the observations

Filtering the time series data

Removing trend and seasonality

Forecasting the future with ARMA and ARIMA models

Graphs

Introduction

Handling graph objects in Python with NetworkX

Using Gephi to visualize graphs

Identifying people whose credit card details were stolen

Identifying those responsible for stealing the credit cards

Natural Language Processing

Introduction

Reading raw text from the Web

Tokenizing and normalizing text

Identifying parts of speech, handling n-grams, and recognizing named entities

Identifying the topic of an article

Identifying the sentence structure

Classifying movies based on their reviews

Discrete Choice Models

Introduction

Preparing a dataset to estimate discrete choice models

Estimating the well-known Multinomial Logit model

Testing for violations of the Independence from Irrelevant Alternatives

Handling IIA violations with the Nested Logit model

Managing sophisticated substitution patterns with the Mixed Logit model

Simulations

Introduction

Using SimPy to simulate the refueling process of a gas station

Simulating out-of-energy occurrences for an electric car

Determining if a population of sheep is in danger of extinction due to a wolf pack

Index

Customer Reviews

5 star

4 star

3 star

2 star

1 star

Removing duplicates

We can safely assume that all the data that lands on our desks is dirty (until proven otherwise). It is a good habit to check whether everything with our data is in order. The first thing I always check for is the duplication of rows.

Getting ready

To follow this recipe, you need to have OpenRefine and virtually any Internet browser installed on your computer.

We assume that you followed the previous recipes and your data is already loaded to OpenRefine and the data types are now representative of what the columns hold. No other prerequisites are required.

How to do it…

First, we assume that within the seven days of property sales, a row is a duplicate if the same address appears twice (or more) in the dataset. It is quite unlikely that the same house is sold twice (or more times) within such a short period of time. Therefore, first, we Blank down the observations if they repeat:

This effects in keeping only the first occurrence of a certain set of observations and blanking the rest (see the fourth row in the following screenshot):

Tip

The Fill down option has the opposite effect—it would fill in the blanks with the values from the row above unless the cell is not blank.

We can now create a Facet by blank that would allow us to quickly select the blanked rows:

Creating such a facet allows us to quickly select all the rows that are blank and remove them from the dataset:

Our dataset now has no duplicate records.

Practical Data Analysis Cookbook

By : Tomasz Drabas

Practical Data Analysis Cookbook

By: Tomasz Drabas

Overview of this book

Related Content you might be interested in

Current Title:

Practical Data Analysis Cookbook

Removing duplicates

Getting ready

How to do it…

Tip