Book Image

Practical Data Science Cookbook

By : Tony Ojeda, Sean Patrick Murphy, Benjamin Bengfort, Abhijit Dasgupta
Book Image

Practical Data Science Cookbook

By: Tony Ojeda, Sean Patrick Murphy, Benjamin Bengfort, Abhijit Dasgupta

Overview of this book

<p>As increasing amounts of data is generated each year, the need to analyze and operationalize it is more important than ever. Companies that know what to do with their data will have a competitive advantage over companies that don't, and this will drive a higher demand for knowledgeable and competent data professionals.</p> <p>Starting with the basics, this book will cover how to set up your numerical programming environment, introduce you to the data science pipeline (an iterative process by which data science projects are completed), and guide you through several data projects in a step-by-step format. By sequentially working through the steps in each chapter, you will quickly familiarize yourself with the process and learn how to apply it to a variety of situations with examples in the two most popular programming languages for data analysis—R and Python.</p>

Related Content you might be interested in

Current Title:

Small book image
Practical Data Science Cookbook
Tony Ojeda | Sean Patrick Murphy | Benjamin Bengfort | Abhijit Dasgupta
Sep, 2014 | 396 pages
No titles found
Table of Contents (18 chapters)
Practical Data Science Cookbook
Practical Data Science Cookbook
Credits
Credits
About the Authors
About the Authors
About the Reviewers
About the Reviewers
www.PacktPub.com
www.PacktPub.com
Preface
Preface
Free Chapter
1
Preparing Your Data Science Environment
Preparing Your Data Science Environment
Introduction
Understanding the data science pipeline
Installing R on Windows, Mac OS X, and Linux
Installing libraries in R and RStudio
Installing Python on Linux and Mac OS X
Installing Python on Windows
Installing the Python data stack on Mac OS X and Linux
Installing extra Python packages
Installing and using virtualenv
2
Driving Visual Analysis with Automobile Data (R)
Driving Visual Analysis with Automobile Data (R)
Introduction
Acquiring automobile fuel efficiency data
Preparing R for your first project
Importing automobile fuel efficiency data into R
Exploring and describing fuel efficiency data
Analyzing automobile fuel efficiency over time
Investigating the makes and models of automobiles
3
Simulating American Football Data (R)
Simulating American Football Data (R)
Introduction
Acquiring and cleaning football data
Analyzing and understanding football data
Constructing indexes to measure offensive and defensive strength
Simulating a single game with outcomes decided by calculations
Simulating multiple games with outcomes decided by calculations
4
Modeling Stock Market Data (R)
Modeling Stock Market Data (R)
Introduction
Acquiring stock market data
Summarizing the data
Cleaning and exploring the data
Generating relative valuations
Screening stocks and analyzing historical prices
5
Visually Exploring Employment Data (R)
Visually Exploring Employment Data (R)
Introduction
Preparing for analysis
Importing employment data into R
Exploring the employment data
Obtaining and merging additional data
Adding geographical information
Extracting state- and county-level wage and employment information
Visualizing geographical distributions of pay
Exploring where the jobs are, by industry
Animating maps for a geospatial time series
Benchmarking performance for some common tasks
6
Creating Application-oriented Analyses Using Tax Data (Python)
Creating Application-oriented Analyses Using Tax Data (Python)
Introduction
Preparing for the analysis of top incomes
Importing and exploring the world's top incomes dataset
Analyzing and visualizing the top income data of the US
Furthering the analysis of the top income groups of the US
Reporting with Jinja2
7
Driving Visual Analyses with Automobile Data (Python)
Driving Visual Analyses with Automobile Data (Python)
Introduction
Getting started with IPython
Exploring IPython Notebook
Preparing to analyze automobile fuel efficiencies
Exploring and describing fuel efficiency data with Python
Analyzing automobile fuel efficiency over time with Python
Investigating the makes and models of automobiles with Python
8
Working with Social Graphs (Python)
Working with Social Graphs (Python)
Introduction
Preparing to work with social networks in Python
Importing networks
Exploring subgraphs within a heroic network
Finding strong ties
Finding key players
Exploring the characteristics of entire networks
Clustering and community detection in social networks
Visualizing graphs
9
Recommending Movies at Scale (Python)
Recommending Movies at Scale (Python)
Introduction
Modeling preference expressions
Understanding the data
Ingesting the movie review data
Finding the highest-scoring movies
Improving the movie-rating system
Measuring the distance between users in the preference space
Computing the correlation between users
Finding the best critic for a user
Predicting movie ratings for users
Collaboratively filtering item by item
Building a nonnegative matrix factorization model
Loading the entire dataset into the memory
Dumping the SVD-based model to the disk
Training the SVD-based model
Testing the SVD-based model
10
Harvesting and Geolocating Twitter Data (Python)
Harvesting and Geolocating Twitter Data (Python)
Introduction
Creating a Twitter application
Understanding the Twitter API v1.1
Determining your Twitter followers and friends
Pulling Twitter user profiles
Making requests without running afoul of Twitter's rate limits
Storing JSON data to the disk
Setting up MongoDB for storing Twitter data
Storing user profiles in MongoDB using PyMongo
Exploring the geographic information available in profiles
Plotting geospatial data in Python
11
Optimizing Numerical Code with NumPy and SciPy (Python)
Optimizing Numerical Code with NumPy and SciPy (Python)
Introduction
Understanding the optimization process
Identifying common performance bottlenecks in code
Reading through the code
Profiling Python code with the Unix time function
Profiling Python code using built-in Python functions
Profiling Python code using IPython's %timeit function
Profiling Python code using line_profiler
Plucking the low-hanging (optimization) fruit
Testing the performance benefits of NumPy
Rewriting simple functions with NumPy
Optimizing the innermost loop with NumPy
Index
Index

Computing the correlation between users

In the previous recipe, we used one out of many possible distance measures to capture the distance between the movie reviews of users. This distance between two specific users is not changed even if there are five or five million other users.

In this recipe, we will compute the correlation between users in the preference space. Like distance metrics, there are many correlation metrics. The most popular of these are Pearson or Spearman correlations or Cosine distance. Unlike distance metrics, the correlation will change depending on the number of users and movies.

Getting ready

We will be continuing the efforts of the previous recipes again, so make sure you understand each one.

How to do it…

The following function implements the computation of the pearson_correlation function for two critics, which are criticA and criticB, and is added to the MovieLens class:

     def pearson_correlation(self, criticA, criticB):
         """
         Returns the Pearson...
Previous Section
End of Section 9
Next Section