Practical Data Analysis Cookbook

Practical Data Analysis Cookbook

By : Tomasz Drabas

Buy this Book

Practical Data Analysis Cookbook

By: Tomasz Drabas

Buy this Book

Overview of this book

Data analysis is the process of systematically applying statistical and logical techniques to describe and illustrate, condense and recap, and evaluate data. Its importance has been most visible in the sector of information and communication technologies. It is an employee asset in almost all economy sectors. This book provides a rich set of independent recipes that dive into the world of data analytics and modeling using a variety of approaches, tools, and algorithms. You will learn the basics of data handling and modeling, and will build your skills gradually toward more advanced topics such as simulations, raw text processing, social interactions analysis, and more. First, you will learn some easy-to-follow practical techniques on how to read, write, clean, reformat, explore, and understand your data—arguably the most time-consuming (and the most important) tasks for any data scientist. In the second section, different independent recipes delve into intermediate topics such as classification, clustering, predicting, and more. With the help of these easy-to-follow recipes, you will also learn techniques that can easily be expanded to solve other real-life problems such as building recommendation engines or predictive models. In the third section, you will explore more advanced topics: from the field of graph theory through natural language processing, discrete choice modeling to simulations. You will also get to expand your knowledge on identifying fraud origin with the help of a graph, scrape Internet websites, and classify movies based on their reviews. By the end of this book, you will be able to efficiently use the vast array of tools that the Python environment has to offer.

Practical Data Analysis Cookbook

Credits

About the Author

Acknowledgments

About the Reviewers

www.PacktPub.com

Preface

Free Chapter

Preparing the Data

Introduction

Reading and writing CSV/TSV files with Python

Reading and writing JSON files with Python

Reading and writing Excel files with Python

Reading and writing XML files with Python

Retrieving HTML pages with pandas

Storing and retrieving from a relational database

Storing and retrieving from MongoDB

Opening and transforming data with OpenRefine

Exploring the data with Open Refine

Removing duplicates

Using regular expressions and GREL to clean up data

Imputing missing observations

Normalizing and standardizing the features

Binning the observations

Encoding categorical variables

Exploring the Data

Introduction

Producing descriptive statistics

Exploring correlations between features

Visualizing the interactions between features

Producing histograms

Creating multivariate charts

Sampling the data

Splitting the dataset into training, cross-validation, and testing

Classification Techniques

Introduction

Testing and comparing the models

Classifying with Naïve Bayes

Using logistic regression as a universal classifier

Utilizing Support Vector Machines as a classification engine

Classifying calls with decision trees

Predicting subscribers with random tree forests

Employing neural networks to classify calls

Clustering Techniques

Introduction

Assessing the performance of a clustering method

Clustering data with k-means algorithm

Finding an optimal number of clusters for k-means

Discovering clusters with mean shift clustering model

Building fuzzy clustering model with c-means

Using hierarchical model to cluster your data

Finding groups of potential subscribers with DBSCAN and BIRCH algorithms

Reducing Dimensions

Introduction

Creating three-dimensional scatter plots to present principal components

Reducing the dimensions using the kernel version of PCA

Using Principal Component Analysis to find things that matter

Finding the principal components in your data using randomized PCA

Extracting the useful dimensions using Linear Discriminant Analysis

Using various dimension reduction techniques to classify calls using the k-Nearest Neighbors classification model

Regression Methods

Introduction

Identifying and tackling multicollinearity

Building Linear Regression model

Using OLS to forecast how much electricity can be produced

Estimating the output of an electric plant using CART

Employing the kNN model in a regression problem

Applying the Random Forest model to a regression analysis

Gauging the amount of electricity a plant can produce using SVMs

Training a Neural Network to predict the output of a power plant

Time Series Techniques

Introduction

Handling date objects in Python

Understanding time series data

Smoothing and transforming the observations

Filtering the time series data

Removing trend and seasonality

Forecasting the future with ARMA and ARIMA models

Graphs

Introduction

Handling graph objects in Python with NetworkX

Using Gephi to visualize graphs

Identifying people whose credit card details were stolen

Identifying those responsible for stealing the credit cards

Natural Language Processing

Introduction

Reading raw text from the Web

Tokenizing and normalizing text

Identifying parts of speech, handling n-grams, and recognizing named entities

Identifying the topic of an article

Identifying the sentence structure

Classifying movies based on their reviews

Discrete Choice Models

Introduction

Preparing a dataset to estimate discrete choice models

Estimating the well-known Multinomial Logit model

Testing for violations of the Independence from Irrelevant Alternatives

Handling IIA violations with the Nested Logit model

Managing sophisticated substitution patterns with the Mixed Logit model

Simulations

Introduction

Using SimPy to simulate the refueling process of a gas station

Simulating out-of-energy occurrences for an electric car

Determining if a population of sheep is in danger of extinction due to a wolf pack

Index

Customer Reviews

5 star

4 star

3 star

2 star

1 star

Encoding categorical variables

The final step on the road to prepare the data for the exploratory phase is to bin categorical variables. Some software packages do this behind the scenes, but it is good to understand when and how to do it.

Any statistical model can accept only numerical data. Categorical data (sometimes can be expressed as digits depending on the context) cannot be used in a model straightaway. To use them, we encode them, that is, give them a unique numerical code. This is to explain when. As for how—you can use the following recipe.

Getting ready

To execute this recipe, you will need the pandas module.

No other prerequisites are required.

How to do it…

Once again, pandas already has a method that does all of this for us (the data_dummy_code.py file):

# dummy code the column with the type of the property
csv_read = pd.get_dummies(
    csv_read,
    prefix='d',
    columns=['type']
)

How it works…

The .get_dummies(...) method converts categorical variables into dummy variables. For example, consider a variable with three different levels:

1  One
2  Two
3  Three

We will need three columns to code it:

1  One  1  0  0
2  Two  0   1  0
3  Three  0  0  1

Sometimes, we can get away with using only two additional columns. However, we can use this trick only if one of the levels is, effectively, null:

1  One  1  0
2  Two  0  1
3  Zero  0  0

The first parameter to the .get_dummies(...) method is the DataFrame. The columns parameter specifies the column (or columns, as we can also pass a list) in the DataFrame to the dummy code. Specifying the prefix, we instruct the method that the names of the new columns generated should have the d_ prefix; in our example, the generated dummy-coded columns will have d_Condo names (as an example). The underscore _ character is default but can also be altered by specifying the prefix_sep parameter.

Tip

For a full list of parameters to the .get_dummies(...) method, see http://pandas.pydata.org/pandas-docs/stable/generated/pandas.get_dummies.html.

Practical Data Analysis Cookbook

By : Tomasz Drabas

Practical Data Analysis Cookbook

By: Tomasz Drabas

Overview of this book

Related Content you might be interested in

Current Title:

Practical Data Analysis Cookbook

Encoding categorical variables

Getting ready

How to do it…

How it works…

Tip