Book Image

Practical Data Science Cookbook, Second Edition - Second Edition

By : Prabhanjan Narayanachar Tattar, Bhushan Purushottam Joshi, Sean Patrick Murphy, ABHIJIT DASGUPTA, Anthony Ojeda
Book Image

Practical Data Science Cookbook, Second Edition - Second Edition

By: Prabhanjan Narayanachar Tattar, Bhushan Purushottam Joshi, Sean Patrick Murphy, ABHIJIT DASGUPTA, Anthony Ojeda

Overview of this book

As increasing amounts of data are generated each year, the need to analyze and create value out of it is more important than ever. Companies that know what to do with their data and how to do it well will have a competitive advantage over companies that don’t. Because of this, there will be an increasing demand for people that possess both the analytical and technical abilities to extract valuable insights from data and create valuable solutions that put those insights to use. Starting with the basics, this book covers how to set up your numerical programming environment, introduces you to the data science pipeline, and guides you through several data projects in a step-by-step format. By sequentially working through the steps in each chapter, you will quickly familiarize yourself with the process and learn how to apply it to a variety of situations with examples using the two most popular programming languages for data analysis—R and Python.

Related Content you might be interested in

Current Title:

Small book image
Practical Data Science Cookbook, Second Edition - Second Edition
Prabhanjan Narayanachar Tattar | Bhushan Purushottam Joshi | Sean Patrick Murphy | ABHIJIT DASGUPTA | Anthony Ojeda
Jun, 2017 | 434 pages
Small book image
Hands-On Ensemble Learning with R
Prabhanjan Narayanachar Tattar
Jul, 2018 | 376 pages
Small book image
Statistical Application Development with R and Python
Prabhanjan Narayanachar Tattar
Aug, 2017 | 432 pages
Small book image
R Data Analysis Cookbook
Shanthi Viswanathan | Kuntal Ganguly | Viswa Viswanathan
Sep, 2017 | 560 pages
Table of Contents (17 chapters)
Title Page
Title Page
Credits
Credits
About the Authors
About the Authors
About the Reviewer
About the Reviewer
www.PacktPub.com
www.PacktPub.com
Preface
Preface
Free Chapter
1
Preparing Your Data Science Environment
Preparing Your Data Science Environment
Understanding the data science pipeline
Installing R on Windows, Mac OS X, and Linux
Installing libraries in R and RStudio
Installing Python on Linux and Mac OS X
Installing Python on Windows
Installing the Python data stack on Mac OS X and Linux
Installing extra Python packages
Installing and using virtualenv
2
Driving Visual Analysis with Automobile Data with R
Driving Visual Analysis with Automobile Data with R
Introduction
Acquiring automobile fuel efficiency data
Preparing R for your first project
Importing automobile fuel efficiency data into R
Exploring and describing fuel efficiency data
Analyzing automobile fuel efficiency over time
Investigating the makes and models of automobiles
3
Creating Application-Oriented Analyses Using Tax Data and Python
Creating Application-Oriented Analyses Using Tax Data and Python
Introduction
Preparing for the analysis of top incomes
Importing and exploring the world's top incomes dataset
Analyzing and visualizing the top income data of the US
Furthering the analysis of the top income groups of the US
Reporting with Jinja2
Repeating the analysis in R
4
Modeling Stock Market Data
Modeling Stock Market Data
Introduction
Acquiring stock market data
Summarizing the data
Cleaning and exploring the data
Generating relative valuations
Screening stocks and analyzing historical prices
5
Visually Exploring Employment Data
Visually Exploring Employment Data
Introduction
Preparing for analysis
Importing employment data into R
Exploring the employment data
Obtaining and merging additional data
Adding geographical information
Extracting state- and county-level wage and employment information
Visualizing geographical distributions of pay
Exploring where the jobs are, by industry
Animating maps for a geospatial time series
Benchmarking performance for some common tasks
6
Driving Visual Analyses with Automobile Data
Driving Visual Analyses with Automobile Data
Introduction
Getting started with IPython
Exploring Jupyter Notebook
Preparing to analyze automobile fuel efficiencies
Exploring and describing fuel efficiency data with Python
Analyzing automobile fuel efficiency over time with Python
Investigating the makes and models of automobiles with Python
7
Working with Social Graphs
Working with Social Graphs
Introduction
Preparing to work with social networks in Python
Importing networks
Exploring subgraphs within a heroic network
Finding strong ties
Finding key players
Exploring the characteristics of entire networks
Clustering and community detection in social networks
Visualizing graphs
Social networks in R
8
Recommending Movies at Scale (Python)
Recommending Movies at Scale (Python)
Introduction
Modeling preference expressions
Understanding the data
Ingesting the movie review data
Finding the highest-scoring movies
Improving the movie-rating system
Measuring the distance between users in the preference space
Computing the correlation between users
Finding the best critic for a user
Predicting movie ratings for users
Collaboratively filtering item by item
Building a non-negative matrix factorization model
Loading the entire dataset into the memory
Dumping the SVD-based model to the disk
Training the SVD-based model
Testing the SVD-based model
9
Harvesting and Geolocating Twitter Data (Python)
Harvesting and Geolocating Twitter Data (Python)
Introduction
Creating a Twitter application
Understanding the Twitter API v1.1
Determining your Twitter followers and friends
Pulling Twitter user profiles
Making requests without running afoul of Twitter's rate limits
Storing JSON data to disk
Setting up MongoDB for storing Twitter data
Storing user profiles in MongoDB using PyMongo
Exploring the geographic information available in profiles
Plotting geospatial data in Python
10
Forecasting New Zealand Overseas Visitors
Forecasting New Zealand Overseas Visitors
Introduction
The ts object
Visualizing time series data
Simple linear regression models
ACF and PACF
ARIMA models
Accuracy measurements
Fitting seasonal ARIMA models
11
German Credit Data Analysis
German Credit Data Analysis
Introduction
Simple data transformations
Visualizing categorical data
Discriminant analysis
Dividing the data and the ROC
Fitting the logistic regression model
Decision trees and rules
Decision tree for german data

Computing the correlation between users

In the previous recipe, we used one out of many possible distance measures to capture the distance between the movie reviews of users. This distance between two specific users is not changed even if there are five or five million other users.

In this recipe, we will compute the correlation between users in the preference space. Like distance metrics, there are many correlation metrics. The most popular of these are Pearson or Spearman correlations or cosine distance. Unlike distance metrics, the correlation will change depending on the number of users and movies.

Getting ready

We will be continuing the efforts of the previous recipes again, so make sure you understand each one.

How to do it...

The following function implements the computation of the pearson_correlation function for two critics, which are criticA and criticB, and it is added to the MovieLens class:

In [15]: def pearson_correlation(self, criticA, criticB, prefs='users'):
...: """
...: Returns...
Previous Section
End of Section 9
Next Section