Python Text Processing with NLTK 2.0 Cookbook

Python Text Processing with NLTK 2.0 Cookbook

By : Jacob Perkins

Buy this Book

Python Text Processing with NLTK 2.0 Cookbook

By: Jacob Perkins

Buy this Book

Overview of this book

Natural Language Processing is used everywhere – in search engines, spell checkers, mobile phones, computer games – even your washing machine. Python's Natural Language Toolkit (NLTK) suite of libraries has rapidly emerged as one of the most efficient tools for Natural Language Processing. You want to employ nothing less than the best techniques in Natural Language Processing – and this book is your answer. Python Text Processing with NLTK 2.0 Cookbook is your handy and illustrative guide, which will walk you through all the Natural Language Processing techniques in a step–by-step manner. It will demystify the advanced features of text analysis and text mining using the comprehensive NLTK suite. This book cuts short the preamble and you dive right into the science of text processing with a practical hands-on approach. Get started off with learning tokenization of text. Get an overview of WordNet and how to use it. Learn the basics as well as advanced features of Stemming and Lemmatization. Discover various ways to replace words with simpler and more common (read: more searched) variants. Create your own corpora and learn to create custom corpus readers for JSON files as well as for data stored in MongoDB. Use and manipulate POS taggers. Transform and normalize parsed chunks to produce a canonical form without changing their meaning. Dig into feature extraction and text classification. Learn how to easily handle huge amounts of data without any loss in efficiency or speed. This book will teach you all that and beyond, in a hands-on learn-by-doing manner. Make yourself an expert in using the NLTK for Natural Language Processing with this handy companion.

Python Text Processing with NLTK 2.0 Cookbook

Credits

About the Author

About the Reviewers

Preface

Free Chapter

Tokenizing Text and WordNet Basics

Introduction

Tokenizing text into sentences

Tokenizing sentences into words

Tokenizing sentences using regular expressions

Filtering stopwords in a tokenized sentence

Looking up synsets for a word in WordNet

Looking up lemmas and synonyms in WordNet

Calculating WordNet synset similarity

Discovering word collocations

Replacing and Correcting Words

Introduction

Stemming words

Lemmatizing words with WordNet

Translating text with Babelfish

Replacing words matching regular expressions

Removing repeating characters

Spelling correction with Enchant

Replacing synonyms

Replacing negations with antonyms

Creating Custom Corpora

Introduction

Setting up a custom corpus

Creating a word list corpus

Creating a part-of-speech tagged word corpus

Creating a chunked phrase corpus

Creating a categorized text corpus

Creating a categorized chunk corpus reader

Lazy corpus loading

Creating a custom corpus view

Creating a MongoDB backed corpus reader

Corpus editing with file locking

Part-of-Speech Tagging

Introduction

Default tagging

Training a unigram part-of-speech tagger

Combining taggers with backoff tagging

Training and combining Ngram taggers

Creating a model of likely word tags

Tagging with regular expressions

Affix tagging

Training a Brill tagger

Training the TnT tagger

Using WordNet for tagging

Tagging proper names

Classifier based tagging

Extracting Chunks

Introduction

Chunking and chinking with regular expressions

Merging and splitting chunks with regular expressions

Expanding and removing chunks with regular expressions

Partial parsing with regular expressions

Training a tagger-based chunker

Classification-based chunking

Extracting named entities

Extracting proper noun chunks

Extracting location chunks

Training a named entity chunker

Transforming Chunks and Trees

Introduction

Filtering insignificant words

Correcting verb forms

Swapping verb phrases

Swapping noun cardinals

Swapping infinitive phrases

Singularizing plural nouns

Chaining chunk transformations

Converting a chunk tree to text

Flattening a deep tree

Creating a shallow tree

Converting tree nodes

Text Classification

Introduction

Bag of Words feature extraction

Training a naive Bayes classifier

Training a decision tree classifier

Training a maximum entropy classifier

Measuring precision and recall of a classifier

Calculating high information words

Combining classifiers with voting

Classifying with multiple binary classifiers

Distributed Processing and Handling Large Datasets

Introduction

Distributed tagging with execnet

Distributed chunking with execnet

Parallel list processing with execnet

Storing a frequency distribution in Redis

Storing a conditional frequency distribution in Redis

Storing an ordered dictionary in Redis

Distributed word scoring with Redis and execnet

Parsing Specific Data

Introduction

Parsing dates and times with Dateutil

Time zone lookup and conversion

Tagging temporal expressions with Timex

Extracting URLs from HTML with lxml

Cleaning and stripping HTML

Converting HTML entities with BeautifulSoup

Detecting and converting character encodings

Penn Treebank Part-of-Speech Tags

Index

Customer Reviews

5 star

4 star

3 star

2 star

1 star

Appendix A. Penn Treebank Part-of-Speech Tags

Following is a table of all the part-of-speech tags that occur in the treebank corpus distributed with NLTK. The tags and counts shown here were acquired using the following code:

>>> from nltk.probability import FreqDist
>>> from nltk.corpus import treebank
>>> fd = FreqDist()
>>> for word, tag in treebank.tagged_words():
...   fd.inc(tag)
>>> fd.items()

The FreqDist fd contains all the counts shown here for every tag in the treebank corpus. You can inspect each tag count individually by doing fd[tag], as in fd['DT']. Punctuation tags are also shown, along with special tags such as -NONE-, which signifies that the part-of-speech tag is unknown. Descriptions of most of the tags can be found at http://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html.

Part-of-speech tag	Frequency of occurrence
#	16
$	724
''	694
,	4,886
-LRB-	120
-NONE-	6,592
-RRB-	126
.	384
:	563
``	712
CC	2,265
CD	3,546
DT	8,165
EX	88
FW	4
IN	9,857
JJ	5,834
JJR	381
JJS	182
LS	13
MD	927
NN	13,166
NNP	9,410
NNPS	244
NNS	6,047
PDT	27
POS	824
PRP	1,716
PRP$	766
RB	2,822
RBR	136
RBS	35
RP	216
SYM	1
TO	2,179
UH	3
VB	2,554
VBD	3,043
VBG	1,460
VBN	2,134
VBP	1,321
VBZ	2,125
WDT	445
WP	241
WP$	14

Python Text Processing with NLTK 2.0 Cookbook

By : Jacob Perkins

Python Text Processing with NLTK 2.0 Cookbook

By: Jacob Perkins

Overview of this book

Related Content you might be interested in

Current Title:

Python Text Processing with NLTK 2.0 Cookbook

Appendix A. Penn Treebank Part-of-Speech Tags