Book Image

Python Text Processing with NLTK 2.0 Cookbook

By : Jacob Perkins
Book Image

Python Text Processing with NLTK 2.0 Cookbook

By: Jacob Perkins

Overview of this book

<p>Natural Language Processing is used everywhere – in search engines, spell checkers, mobile phones, computer games – even your washing machine. Python's Natural Language Toolkit (NLTK) suite of libraries has rapidly emerged as one of the most efficient tools for Natural Language Processing. You want to employ nothing less than the best techniques in Natural Language Processing – and this book is your answer.<br /><br /><em>Python Text Processing with NLTK 2.0 Cookbook</em> is your handy and illustrative guide, which will walk you through all the Natural Language Processing techniques in a step–by-step manner. It will demystify the advanced features of text analysis and text mining using the comprehensive NLTK suite.<br /><br />This book cuts short the preamble and you dive right into the science of text processing with a practical hands-on approach.<br /><br />Get started off with learning tokenization of text. Get an overview of WordNet and how to use it. Learn the basics as well as advanced features of Stemming and Lemmatization. Discover various ways to replace words with simpler and more common (read: more searched) variants. Create your own corpora and learn to create custom corpus readers for JSON files as well as for data stored in MongoDB. Use and manipulate POS taggers. Transform and normalize parsed chunks to produce a canonical form without changing their meaning. Dig into feature extraction and text classification. Learn how to easily handle huge amounts of data without any loss in efficiency or speed.<br /><br />This book will teach you all that and beyond, in a hands-on learn-by-doing manner. Make yourself an expert in using the NLTK for Natural Language Processing with this handy companion.</p>

Related Content you might be interested in

Current Title:

Small book image
Python Text Processing with NLTK 2.0 Cookbook
Jacob Perkins
Nov, 2010 | 272 pages
No titles found
Table of Contents (16 chapters)
Python Text Processing with NLTK 2.0 Cookbook
Python Text Processing with NLTK 2.0 Cookbook
Credits
Credits
About the Author
About the Author
About the Reviewers
About the Reviewers
Preface
Preface
Free Chapter
1
Tokenizing Text and WordNet Basics
Tokenizing Text and WordNet Basics
Introduction
Tokenizing text into sentences
Tokenizing sentences into words
Tokenizing sentences using regular expressions
Filtering stopwords in a tokenized sentence
Looking up synsets for a word in WordNet
Looking up lemmas and synonyms in WordNet
Calculating WordNet synset similarity
Discovering word collocations
2
Replacing and Correcting Words
Replacing and Correcting Words
Introduction
Stemming words
Lemmatizing words with WordNet
Translating text with Babelfish
Replacing words matching regular expressions
Removing repeating characters
Spelling correction with Enchant
Replacing synonyms
Replacing negations with antonyms
3
Creating Custom Corpora
Creating Custom Corpora
Introduction
Setting up a custom corpus
Creating a word list corpus
Creating a part-of-speech tagged word corpus
Creating a chunked phrase corpus
Creating a categorized text corpus
Creating a categorized chunk corpus reader
Lazy corpus loading
Creating a custom corpus view
Creating a MongoDB backed corpus reader
Corpus editing with file locking
4
Part-of-Speech Tagging
Part-of-Speech Tagging
Introduction
Default tagging
Training a unigram part-of-speech tagger
Combining taggers with backoff tagging
Training and combining Ngram taggers
Creating a model of likely word tags
Tagging with regular expressions
Affix tagging
Training a Brill tagger
Training the TnT tagger
Using WordNet for tagging
Tagging proper names
Classifier based tagging
5
Extracting Chunks
Extracting Chunks
Introduction
Chunking and chinking with regular expressions
Merging and splitting chunks with regular expressions
Expanding and removing chunks with regular expressions
Partial parsing with regular expressions
Training a tagger-based chunker
Classification-based chunking
Extracting named entities
Extracting proper noun chunks
Extracting location chunks
Training a named entity chunker
6
Transforming Chunks and Trees
Transforming Chunks and Trees
Introduction
Filtering insignificant words
Correcting verb forms
Swapping verb phrases
Swapping noun cardinals
Swapping infinitive phrases
Singularizing plural nouns
Chaining chunk transformations
Converting a chunk tree to text
Flattening a deep tree
Creating a shallow tree
Converting tree nodes
7
Text Classification
Text Classification
Introduction
Bag of Words feature extraction
Training a naive Bayes classifier
Training a decision tree classifier
Training a maximum entropy classifier
Measuring precision and recall of a classifier
Calculating high information words
Combining classifiers with voting
Classifying with multiple binary classifiers
8
Distributed Processing and Handling Large Datasets
Distributed Processing and Handling Large Datasets
Introduction
Distributed tagging with execnet
Distributed chunking with execnet
Parallel list processing with execnet
Storing a frequency distribution in Redis
Storing a conditional frequency distribution in Redis
Storing an ordered dictionary in Redis
Distributed word scoring with Redis and execnet
9
Parsing Specific Data
Parsing Specific Data
Introduction
Parsing dates and times with Dateutil
Time zone lookup and conversion
Tagging temporal expressions with Timex
Extracting URLs from HTML with lxml
Cleaning and stripping HTML
Converting HTML entities with BeautifulSoup
Detecting and converting character encodings
Penn Treebank Part-of-Speech Tags
Penn Treebank Part-of-Speech Tags
Index
Index

Storing a conditional frequency distribution in Redis

The nltk.probability.ConditionalFreqDist class is a container for FreqDist instances, with one FreqDist per condition. It is used to count frequencies that are dependent on another condition, such as another word or a class label. We used this class in the Calculating high information words recipe in Chapter 7, Text Classification. Here, we'll create an API-compatible class on top of Redis using the RedisHashFreqDist from the previous recipe.

Getting ready

As in the previous recipe, you'll need to have Redis and redis-py installed with an instance of redis-server running.

How to do it...

We define a RedisConditionalHashFreqDist class in redisprob.py that extends nltk.probability.ConditionalFreqDist and overrides the __contains__() and __getitem__() methods. We then override __getitem__() so we can create an instance of RedisHashFreqDist instead of a FreqDist, and override __contains__() so we can call encode_key() from the rediscollections...

Previous Section
End of Section 7
Next Section