Book Image

Mastering Natural Language Processing with Python

By : Deepti Chopra, Nisheeth Joshi, Iti Mathur
Book Image

Mastering Natural Language Processing with Python

By: Deepti Chopra, Nisheeth Joshi, Iti Mathur

Overview of this book

<p>Natural Language Processing is one of the fields of computational linguistics and artificial intelligence that is concerned with human-computer interaction. It provides a seamless interaction between computers and human beings and gives computers the ability to understand human speech with the help of machine learning.</p> <p>This book will give you expertise on how to employ various NLP tasks in Python, giving you an insight into the best practices when designing and building NLP-based applications using Python. It will help you become an expert in no time and assist you in creating your own NLP projects using NLTK.</p> <p>You will sequentially be guided through applying machine learning tools to develop various models. We’ll give you clarity on how to create training data and how to implement major NLP applications such as Named Entity Recognition, Question Answering System, Discourse Analysis, Transliteration, Word Sense disambiguation, Information Retrieval, Sentiment Analysis, Text Summarization, and Anaphora Resolution.</p>

Related Content you might be interested in

Current Title:

Small book image
Mastering Natural Language Processing with Python
Deepti Chopra | Nisheeth Joshi | Iti Mathur
Jun, 2016 | 238 pages
No titles found
Table of Contents (17 chapters)
Mastering Natural Language Processing with Python
Mastering Natural Language Processing with Python
Credits
Credits
About the Authors
About the Authors
About the Reviewer
About the Reviewer
www.PacktPub.com
www.PacktPub.com
Preface
Preface
Free Chapter
1
Working with Strings
Working with Strings
Tokenization
Normalization
Substituting and correcting tokens
Applying Zipf's law to text
Similarity measures
Summary
2
Statistical Language Modeling
Statistical Language Modeling
Understanding word frequency
Applying smoothing on the MLE model
Develop a back-off mechanism for MLE
Applying interpolation on data to get mix and match
Evaluate a language model through perplexity
Applying metropolis hastings in modeling languages
Applying Gibbs sampling in language processing
Summary
3
Morphology – Getting Our Feet Wet
Morphology – Getting Our Feet Wet
Introducing morphology
Understanding stemmer
Understanding lemmatization
Developing a stemmer for non-English language
Morphological analyzer
Morphological generator
Search engine
Summary
4
Parts-of-Speech Tagging – Identifying Words
Parts-of-Speech Tagging – Identifying Words
Introducing parts-of-speech tagging
Creating POS-tagged corpora
Selecting a machine learning algorithm
Statistical modeling involving the n-gram approach
Developing a chunker using pos-tagged corpora
Summary
5
Parsing – Analyzing Training Data
Parsing – Analyzing Training Data
Introducing parsing
Treebank construction
Extracting Context Free Grammar (CFG) rules from Treebank
Creating a probabilistic Context Free Grammar from CFG
CYK chart parsing algorithm
Earley chart parsing algorithm
Summary
6
Semantic Analysis – Meaning Matters
Semantic Analysis – Meaning Matters
Introducing semantic analysis
Generation of the synset id from Wordnet
Disambiguating senses using Wordnet
Summary
7
Sentiment Analysis – I Am Happy
Sentiment Analysis – I Am Happy
Introducing sentiment analysis
Summary
8
Information Retrieval – Accessing Information
Information Retrieval – Accessing Information
Introducing information retrieval
Vector space scoring and query operator interaction
Developing an IR system using latent semantic indexing
Text summarization
Question-answering system
Summary
9
Discourse Analysis – Knowing Is Believing
Discourse Analysis – Knowing Is Believing
Introducing discourse analysis
Summary
10
Evaluation of NLP Systems – Analyzing Performance
Evaluation of NLP Systems – Analyzing Performance
The need for evaluation of NLP systems
Evaluation of IR system
Metrics for error identification
Metrics based on lexical matching
Metrics based on syntactic matching
Metrics using shallow semantic matching
Summary
Index
Index

Developing a stemmer for non-English language

Polyglot is a software that is used to provide models called morfessor models that are used to obtain morphemes from tokens. The Morpho project's goal is to create unsupervised data-driven processes. The main aim of the Morpho project is to focus on the creation of morphemes, which is the smallest unit of syntax. Morphemes play an important role in natural language processing. Morphemes are useful in automatic recognition and the creation of language. With the help of the vocabulary dictionaries of Polyglot, morfessor models on the 50,000 tokens of different languages were used.

Let's see the code for obtaining the language table using polyglot:

from polyglot.downloader import downloader
print(downloader.supported_languages_table("morph2"))

The output obtained from preceding code is the languages listed here:

The necessary models can be downloaded using the following code:

%%bash
polyglot download morph2.en morph2.ar

[polyglot_data] Downloading package...
Previous Section
End of Section 5
Next Section