Book Image

Practical Data Analysis

By : Hector Cuesta
Book Image

Practical Data Analysis

By: Hector Cuesta

Overview of this book

Plenty of small businesses face big amounts of data but lack the internal skills to support quantitative analysis. Understanding how to harness the power of data analysis using the latest open source technology can lead them to providing better customer service, the visualization of customer needs, or even the ability to obtain fresh insights about the performance of previous products. Practical Data Analysis is a book ideal for home and small business users who want to slice and dice the data they have on hand with minimum hassle.Practical Data Analysis is a hands-on guide to understanding the nature of your data and turn it into insight. It will introduce you to the use of machine learning techniques, social networks analytics, and econometrics to help your clients get insights about the pool of data they have at hand. Performing data preparation and processing over several kinds of data such as text, images, graphs, documents, and time series will also be covered.Practical Data Analysis presents a detailed exploration of the current work in data analysis through self-contained projects. First you will explore the basics of data preparation and transformation through OpenRefine. Then you will get started with exploratory data analysis using the D3js visualization framework. You will also be introduced to some of the machine learning techniques such as, classification, regression, and clusterization through practical projects such as spam classification, predicting gold prices, and finding clusters in your Facebook friends' network. You will learn how to solve problems in text classification, simulation, time series forecast, social media, and MapReduce through detailed projects. Finally you will work with large amounts of Twitter data using MapReduce to perform a sentiment analysis implemented in Python and MongoDB. Practical Data Analysis contains a combination of carefully selected algorithms and data scrubbing that enables you to turn your data into insight.

Related Content you might be interested in

Current Title:

Small book image
Practical Data Analysis
Hector Cuesta
Oct, 2013 | 360 pages
No titles found
Table of Contents (24 chapters)
Practical Data Analysis
Practical Data Analysis
Credits
Credits
Foreword
Foreword
About the Author
About the Author
Acknowledgments
Acknowledgments
About the Reviewers
About the Reviewers
www.PacktPub.com
www.PacktPub.com
Preface
Preface
Free Chapter
1
Getting Started
Getting Started
Computer science
Artificial intelligence (AI)
Machine Learning (ML)
Statistics
Mathematics
Knowledge domain
Data, information, and knowledge
The nature of data
The data analysis process
Quantitative versus qualitative data analysis
Importance of data visualization
What about big data?
Summary
2
Working with Data
Working with Data
Datasource
Data scrubbing
Data formats
Getting started with OpenRefine
Summary
3
Data Visualization
Data Visualization
Data-Driven Documents (D3)
Getting started with D3.js
Interaction and animation
Summary
4
Text Classification
Text Classification
Learning and classification
Bayesian classification
E-mail subject line tester
The algorithm
Classifier accuracy
Summary
5
Similarity-based Image Retrieval
Similarity-based Image Retrieval
Image similarity search
Dynamic time warping (DTW)
Processing the image dataset
Implementing DTW
Analyzing the results
Summary
6
Simulation of Stock Prices
Simulation of Stock Prices
Financial time series
Random walk simulation
Monte Carlo methods
Generating random numbers
Implementation in D3.js
Summary
7
Predicting Gold Prices
Predicting Gold Prices
Working with the time series data
Smoothing the time series
The data – historical gold prices
Nonlinear regression
Summary
8
Working with Support Vector Machines
Working with Support Vector Machines
Understanding the multivariate dataset
Dimensionality reduction
Getting started with support vector machine
Summary
9
Modeling Infectious Disease with Cellular Automata
Modeling Infectious Disease with Cellular Automata
Introduction to epidemiology
The epidemic models
Modeling with cellular automata
Simulation of the SIRS model in CA with D3.js
Summary
10
Working with Social Graphs
Working with Social Graphs
Structure of a graph
Social Networks Analysis
Acquiring my Facebook graph
Representing graphs with Gephi
Statistical analysis
Degree distribution
Transforming GDF to JSON
Graph visualization with D3.js
Summary
11
Sentiment Analysis of Twitter Data
Sentiment Analysis of Twitter Data
The anatomy of Twitter data
Using OAuth to access Twitter API
Getting started with Twython
Sentiment classification
Getting started with Natural Language Toolkit (NLTK)
Summary
12
Data Processing and Aggregation with MongoDB
Data Processing and Aggregation with MongoDB
Getting started with MongoDB
Data preparation
Group
The aggregation framework
Summary
13
Working with MapReduce
Working with MapReduce
MapReduce overview
Programming model
Using MapReduce with MongoDB
Filtering the input collection
Grouping and aggregation
Word cloud visualization of the most common positive words in tweets
Summary
14
Online Data Analysis with IPython and Wakari
Online Data Analysis with IPython and Wakari
Getting started with Wakari
Getting started with IPython Notebook
Introduction to image processing with PIL
Getting started with Pandas
Multiprocessing with IPython
Sharing your Notebook
Summary
Setting Up the Infrastructure
Setting Up the Infrastructure
Installing and running Python 3
Installing and running NumPy
Installing and running SciPy
Installing and running mlpy
Installing and running OpenRefine
Installing and running MongoDB
Installing and running UMongo
Installing and running Gephi
Index
Index

The algorithm

We use the list_words() function to get a list of unique words which are more than three-characters long and in lower case:

def list_words(text):
  words = []
  words_tmp = text.lower().split()
  for w in words_tmp:
    if w not in words and len(w) > 3:
      words.append(w)
  return words

Tip

For a more advanced term-document matrix, we can use Python's textmining package from https://pypi.python.org/pypi/textmining/1.0.

The training() function creates variables to store the data needed for the classification. The c_words variable is a dictionary with the unique words and its number of occurrences in the text (frequency) by category. The c_categories variable stores a dictionary of each category and its number of texts. Finally, c_text and c_total_words store the total count of texts and words respectively:

def training(texts):
  c_words ={}
  c_categories ={}
  c_texts = 0
  c_total_words =0
  #add the classes to the categories
  for t in texts:
    c_texts = c_texts + 1
 ...
Previous Section
End of Section 5
Next Section