Book Image

Mastering Natural Language Processing with Python

By : Deepti Chopra, Nisheeth Joshi, Iti Mathur
Book Image

Mastering Natural Language Processing with Python

By: Deepti Chopra, Nisheeth Joshi, Iti Mathur

Overview of this book

<p>Natural Language Processing is one of the fields of computational linguistics and artificial intelligence that is concerned with human-computer interaction. It provides a seamless interaction between computers and human beings and gives computers the ability to understand human speech with the help of machine learning.</p> <p>This book will give you expertise on how to employ various NLP tasks in Python, giving you an insight into the best practices when designing and building NLP-based applications using Python. It will help you become an expert in no time and assist you in creating your own NLP projects using NLTK.</p> <p>You will sequentially be guided through applying machine learning tools to develop various models. We’ll give you clarity on how to create training data and how to implement major NLP applications such as Named Entity Recognition, Question Answering System, Discourse Analysis, Transliteration, Word Sense disambiguation, Information Retrieval, Sentiment Analysis, Text Summarization, and Anaphora Resolution.</p>

Related Content you might be interested in

Current Title:

Small book image
Mastering Natural Language Processing with Python
Deepti Chopra | Nisheeth Joshi | Iti Mathur
Jun, 2016 | 238 pages
No titles found
Table of Contents (17 chapters)
Mastering Natural Language Processing with Python
Mastering Natural Language Processing with Python
Credits
Credits
About the Authors
About the Authors
About the Reviewer
About the Reviewer
www.PacktPub.com
www.PacktPub.com
Preface
Preface
Free Chapter
1
Working with Strings
Working with Strings
Tokenization
Normalization
Substituting and correcting tokens
Applying Zipf's law to text
Similarity measures
Summary
2
Statistical Language Modeling
Statistical Language Modeling
Understanding word frequency
Applying smoothing on the MLE model
Develop a back-off mechanism for MLE
Applying interpolation on data to get mix and match
Evaluate a language model through perplexity
Applying metropolis hastings in modeling languages
Applying Gibbs sampling in language processing
Summary
3
Morphology – Getting Our Feet Wet
Morphology – Getting Our Feet Wet
Introducing morphology
Understanding stemmer
Understanding lemmatization
Developing a stemmer for non-English language
Morphological analyzer
Morphological generator
Search engine
Summary
4
Parts-of-Speech Tagging – Identifying Words
Parts-of-Speech Tagging – Identifying Words
Introducing parts-of-speech tagging
Creating POS-tagged corpora
Selecting a machine learning algorithm
Statistical modeling involving the n-gram approach
Developing a chunker using pos-tagged corpora
Summary
5
Parsing – Analyzing Training Data
Parsing – Analyzing Training Data
Introducing parsing
Treebank construction
Extracting Context Free Grammar (CFG) rules from Treebank
Creating a probabilistic Context Free Grammar from CFG
CYK chart parsing algorithm
Earley chart parsing algorithm
Summary
6
Semantic Analysis – Meaning Matters
Semantic Analysis – Meaning Matters
Introducing semantic analysis
Generation of the synset id from Wordnet
Disambiguating senses using Wordnet
Summary
7
Sentiment Analysis – I Am Happy
Sentiment Analysis – I Am Happy
Introducing sentiment analysis
Summary
8
Information Retrieval – Accessing Information
Information Retrieval – Accessing Information
Introducing information retrieval
Vector space scoring and query operator interaction
Developing an IR system using latent semantic indexing
Text summarization
Question-answering system
Summary
9
Discourse Analysis – Knowing Is Believing
Discourse Analysis – Knowing Is Believing
Introducing discourse analysis
Summary
10
Evaluation of NLP Systems – Analyzing Performance
Evaluation of NLP Systems – Analyzing Performance
The need for evaluation of NLP systems
Evaluation of IR system
Metrics for error identification
Metrics based on lexical matching
Metrics based on syntactic matching
Metrics using shallow semantic matching
Summary
Index
Index

Search engine

PyStemmer 1.0.1 consists of Snowball stemming algorithms that are used for performing information retrieval tasks and for constructing a search engine. It consists of the Porter stemming algorithm and many other stemming algorithms that are useful for performing stemming and information retrieval tasks in many languages, including many European languages.

We can construct a vector space search engine by converting the texts into vectors.

The following are the steps involved in constructing a vector space search engine:

  1. Consider the following code for the removal of stopwords and tokenization:

    A stemmer is a program that accepts words and converts them into stems. Tokens that have the same stem have nearly the same meanings. Stopwords are also eliminated from a text.

    def eliminatestopwords(self,list):
"""
Eliminate words which occur often and have not much significance from context point of view.
"""
return[ word for word in list if word not in self.stopwords ]

def tokenize(self...
Previous Section
End of Section 8
Next Section