Book Image

Practical Data Analysis - Second Edition

By : Hector Cuesta, Dr. Sampath Kumar
Book Image

Practical Data Analysis - Second Edition

By: Hector Cuesta, Dr. Sampath Kumar

Overview of this book

Beyond buzzwords like Big Data or Data Science, there are a great opportunities to innovate in many businesses using data analysis to get data-driven products. Data analysis involves asking many questions about data in order to discover insights and generate value for a product or a service. This book explains the basic data algorithms without the theoretical jargon, and you’ll get hands-on turning data into insights using machine learning techniques. We will perform data-driven innovation processing for several types of data such as text, Images, social network graphs, documents, and time series, showing you how to implement large data processing with MongoDB and Apache Spark.
Table of Contents (21 chapters)
Practical Data Analysis - Second Edition
Credits
About the Authors
About the Reviewers
www.PacktPub.com
Preface

Classifier accuracy


Now we need to test our classifier with a bigger test set; in this case, we will randomly select 100 subjects: 50 spam and 50 not spam. Finally, we will count how many times the classifier chose the correct category:

with open("test.csv") as f: 
    correct = 0 
    tests = csv.reader(f) 
    for subject in test: 
          clase = classifier(subject[0],w,c,t,tw) 
          if clase[1] =subject[1]: 
      correct += 1 
     print("Efficiency : {0} of 100".format(correct)) 

In this case, the Efficiency is 82 percent:

>>> Efficiency: 82 of 100

Tip

We can use an out of the box implementation of the Naive Bayes classifier, like the NaiveBayesClassifier function in the NLTK package for Python. NLTK provides a very powerful natural language toolkit and we can download it from http://nltk.org/.

In Chapter 1, Getting Started, we presented a more sophisticated version of the Naïve Bayes classifier to perform a sentiment analysis.

In this case, we will find an optimal size...