Book Image

Natural Language Processing Fundamentals

By : Sohom Ghosh, Dwight Gunning
Book Image

Natural Language Processing Fundamentals

By: Sohom Ghosh, Dwight Gunning

Overview of this book

If NLP hasn't been your forte, Natural Language Processing Fundamentals will make sure you set off to a steady start. This comprehensive guide will show you how to effectively use Python libraries and NLP concepts to solve various problems. You'll be introduced to natural language processing and its applications through examples and exercises. This will be followed by an introduction to the initial stages of solving a problem, which includes problem definition, getting text data, and preparing it for modeling. With exposure to concepts like advanced natural language processing algorithms and visualization techniques, you'll learn how to create applications that can extract information from unstructured data and present it as impactful visuals. Although you will continue to learn NLP-based techniques, the focus will gradually shift to developing useful applications. In these sections, you'll understand how to apply NLP techniques to answer questions as can be used in chatbots. By the end of this book, you'll be able to accomplish a varied range of assignments ranging from identifying the most suitable type of NLP task for solving a problem to using a tool like spacy or gensim for performing sentiment analysis. The book will easily equip you with the knowledge you need to build applications that interpret human language.

Related Content you might be interested in

Current Title:

Small book image
Natural Language Processing Fundamentals
Sohom Ghosh | Dwight Gunning
Mar, 2019 | 374 pages
Small book image
The Handbook of NLP with Gensim
Chris Kuo
Oct, 2023 | 310 pages
Small book image
Hands-On Python Natural Language Processing
Mayank Rasu | Aman Kedia
Jun, 2020 | 316 pages
Small book image
Python Natural Language Processing Cookbook
Zhenya Antić
Mar, 2021 | 284 pages
Table of Contents (10 chapters)
Preface
Preface
About the Book
Free Chapter
1
1. Introduction to Natural Language Processing
1. Introduction to Natural Language Processing
Introduction
History of NLP
Text Analytics and NLP
Various Steps in NLP
Kick Starting an NLP Project
Summary
2
2. Basic Feature Extraction Methods
2. Basic Feature Extraction Methods
Introduction
Types of Data
Cleaning Text Data
Feature Extraction from Texts
Feature Engineering
Summary
3
3. Developing a Text classifier
3. Developing a Text classifier
Introduction
Machine Learning
Developing a Text Classifier
Building Pipelines for NLP Projects
Saving and Loading Models
Summary
4
4. Collecting Text Data from the Web
4. Collecting Text Data from the Web
Introduction
Collecting Data by Scraping Web Pages
Requesting Content from Web Pages
Dealing with Semi-Structured Data
Summary
5
5. Topic Modeling
5. Topic Modeling
Introduction
Topic Discovery
Topic Modeling Algorithms
Summary
6
6. Text Summarization and Text Generation
6. Text Summarization and Text Generation
Introduction
What is Automated Text Summarization?
High-Level View of Text Summarization
TextRank
Summarizing Text Using Gensim
Summarizing Text Using Word Frequency
Generating Text with Markov Chains
Summary
7
7. Vector Representation
7. Vector Representation
Introduction
Vector Definition
Why Vector Representations?
Summary
8
8. Sentiment Analysis
8. Sentiment Analysis
Introduction
Why is Sentiment Analysis Required?
Growth of Sentiment Analysis
Tools Used for Sentiment Analysis
TextBlob
Understanding Data for Sentiment Analysis
Training Sentiment Models
Summary
Appendix
Appendix
1. Introduction to Natural Language Processing
2. Basic Feature Extraction Methods
3. Developing a Text classifier
4. Collecting Text Data from the Web
5. Topic Modeling
6. Text Summarization and Text Generation
7. Vector Representation
8. Sentiment Analysis

3. Developing a Text classifier

Activity 5: Developing End-to-End Text Classifiers

Solution

Let's build an end-to-end classifier that helps classify Wikipedia comments. Follow these steps to implement this activity:

  1. Open a Jupyter notebook.
  2. Insert a new cell and add the following code to import the necessary packages:
    import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline
import re
import string
from nltk import word_tokenize
from nltk.corpus import stopwords
from nltk.stem import WordNetLemmatizer
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.model_selection import train_test_split
from pylab import *
import nltk
import warnings
warnings.filterwarnings('ignore')
from sklearn.metrics import accuracy_score,roc_curve,classification_report,confusion_matrix,precision_recall_curve,auc
  3. In this step, we will read a data file. It has two columns: comment_text and toxic. The comment_text column...
Previous Section
End of Section
Next Section