Book Image

Python 3 Text Processing with NLTK 3 Cookbook

By : Jacob Perkins
Book Image

Python 3 Text Processing with NLTK 3 Cookbook

By: Jacob Perkins

Overview of this book

Table of Contents (17 chapters)
Python 3 Text Processing with NLTK 3 Cookbook
Credits
About the Author
About the Reviewers
www.PacktPub.com
Preface
Penn Treebank Part-of-speech Tags
Index

Distributed word scoring with Redis and execnet


We can use Redis and execnet together to do distributed word scoring. In the Calculating high information words recipe in Chapter 7, Text Classification, we calculated the information gain of each word in the movie_reviews corpus using a FreqDist and ConditionalFreqDist. Now that we have Redis, we can do the same thing using a RedisHashFreqDist and a RedisConditionalHashFreqDist, and then store the scores in a RedisOrderedDict. We can use execnet to distribute the counting in order to get a better performance out of Redis.

Getting ready

Redis, redis-py, and execnet must be installed, and an instance of redis-server must be running on localhost.

How to do it...

We start by getting a list of (label, words) tuples for each label in the movie_reviews corpus (which only has pos and neg labels). Then, we get the word_scores using score_words() from the dist_featx module. The word_scores function is an instance of RedisOrderedDict, and we can see that...