Book Image

Python 3 Text Processing with NLTK 3 Cookbook

By : Jacob Perkins
Book Image

Python 3 Text Processing with NLTK 3 Cookbook

By: Jacob Perkins

Overview of this book

Related Content you might be interested in

Current Title:

Small book image
Python 3 Text Processing with NLTK 3 Cookbook
Jacob Perkins
Aug, 2014 | 304 pages
No titles found
Table of Contents (17 chapters)
Python 3 Text Processing with NLTK 3 Cookbook
Python 3 Text Processing with NLTK 3 Cookbook
Credits
Credits
About the Author
About the Author
About the Reviewers
About the Reviewers
www.PacktPub.com
www.PacktPub.com
Preface
Preface
Free Chapter
1
Tokenizing Text and WordNet Basics
Tokenizing Text and WordNet Basics
Introduction
Tokenizing text into sentences
Tokenizing sentences into words
Tokenizing sentences using regular expressions
Training a sentence tokenizer
Filtering stopwords in a tokenized sentence
Looking up Synsets for a word in WordNet
Looking up lemmas and synonyms in WordNet
Calculating WordNet Synset similarity
Discovering word collocations
2
Replacing and Correcting Words
Replacing and Correcting Words
Introduction
Stemming words
Lemmatizing words with WordNet
Replacing words matching regular expressions
Removing repeating characters
Spelling correction with Enchant
Replacing synonyms
Replacing negations with antonyms
3
Creating Custom Corpora
Creating Custom Corpora
Introduction
Setting up a custom corpus
Creating a wordlist corpus
Creating a part-of-speech tagged word corpus
Creating a chunked phrase corpus
Creating a categorized text corpus
Creating a categorized chunk corpus reader
Lazy corpus loading
Creating a custom corpus view
Creating a MongoDB-backed corpus reader
Corpus editing with file locking
4
Part-of-speech Tagging
Part-of-speech Tagging
Introduction
Default tagging
Training a unigram part-of-speech tagger
Combining taggers with backoff tagging
Training and combining ngram taggers
Creating a model of likely word tags
Tagging with regular expressions
Affix tagging
Training a Brill tagger
Training the TnT tagger
Using WordNet for tagging
Tagging proper names
Classifier-based tagging
Training a tagger with NLTK-Trainer
5
Extracting Chunks
Extracting Chunks
Introduction
Chunking and chinking with regular expressions
Merging and splitting chunks with regular expressions
Expanding and removing chunks with regular expressions
Partial parsing with regular expressions
Training a tagger-based chunker
Classification-based chunking
Extracting named entities
Extracting proper noun chunks
Extracting location chunks
Training a named entity chunker
Training a chunker with NLTK-Trainer
6
Transforming Chunks and Trees
Transforming Chunks and Trees
Introduction
Filtering insignificant words from a sentence
Correcting verb forms
Swapping verb phrases
Swapping noun cardinals
Swapping infinitive phrases
Singularizing plural nouns
Chaining chunk transformations
Converting a chunk tree to text
Flattening a deep tree
Creating a shallow tree
Converting tree labels
7
Text Classification
Text Classification
Introduction
Bag of words feature extraction
Training a Naive Bayes classifier
Training a decision tree classifier
Training a maximum entropy classifier
Training scikit-learn classifiers
Measuring precision and recall of a classifier
Calculating high information words
Combining classifiers with voting
Classifying with multiple binary classifiers
Training a classifier with NLTK-Trainer
8
Distributed Processing and Handling Large Datasets
Distributed Processing and Handling Large Datasets
Introduction
Distributed tagging with execnet
Distributed chunking with execnet
Parallel list processing with execnet
Storing a frequency distribution in Redis
Storing a conditional frequency distribution in Redis
Storing an ordered dictionary in Redis
Distributed word scoring with Redis and execnet
9
Parsing Specific Data Types
Parsing Specific Data Types
Introduction
Parsing dates and times with dateutil
Timezone lookup and conversion
Extracting URLs from HTML with lxml
Cleaning and stripping HTML
Converting HTML entities with BeautifulSoup
Detecting and converting character encodings
Penn Treebank Part-of-speech Tags
Penn Treebank Part-of-speech Tags
Index
Index

Introduction

In this chapter, we will go over various word replacement and correction techniques. The recipes cover the gamut of linguistic compression, spelling correction, and text normalization. All of these methods can be very useful for preprocessing text before search indexing, document classification, and text analysis.

Previous Section
End of Section 2
Next Section