Book Image

Python Natural Language Processing Cookbook

By : Zhenya Antić
Book Image

Python Natural Language Processing Cookbook

By: Zhenya Antić

Overview of this book

Python is the most widely used language for natural language processing (NLP) thanks to its extensive tools and libraries for analyzing text and extracting computer-usable data. This book will take you through a range of techniques for text processing, from basics such as parsing the parts of speech to complex topics such as topic modeling, text classification, and visualization. Starting with an overview of NLP, the book presents recipes for dividing text into sentences, stemming and lemmatization, removing stopwords, and parts of speech tagging to help you to prepare your data. You’ll then learn ways of extracting and representing grammatical information, such as dependency parsing and anaphora resolution, discover different ways of representing the semantics using bag-of-words, TF-IDF, word embeddings, and BERT, and develop skills for text classification using keywords, SVMs, LSTMs, and other techniques. As you advance, you’ll also see how to extract information from text, implement unsupervised and supervised techniques for topic modeling, and perform topic modeling of short texts, such as tweets. Additionally, the book shows you how to develop chatbots using NLTK and Rasa and visualize text data. By the end of this NLP book, you’ll have developed the skills to use a powerful set of tools for text processing.

Related Content you might be interested in

Current Title:

Small book image
Python Natural Language Processing Cookbook
Zhenya Antić
Mar, 2021 | 284 pages
Small book image
Mastering spaCy
Duygu Altınok
Jul, 2021 | 356 pages
Small book image
Natural Language Processing with Python Quick Start Guide
Nirant Kasliwal
Nov, 2018 | 182 pages
Small book image
Natural Language Processing and Computational Linguistics
Bhargav SrinivasaDesikan
Jun, 2018 | 306 pages
Table of Contents (10 chapters)
Preface
Preface
Who this book is for
What this book covers
To get the most out of this book
Download the color images
Conventions used
Sections
Get in touch
Reviews
1
Chapter 1: Learning NLP Basics
Chapter 1: Learning NLP Basics
Technical requirements
Dividing text into sentences
Dividing sentences into words – tokenization
Parts of speech tagging
Word stemming
Combining similar words – lemmatization
Removing stopwords
Free Chapter
2
Chapter 2: Playing with Grammar
Chapter 2: Playing with Grammar
Technical requirements
Counting nouns – plural and singular nouns
Getting the dependency parse
Splitting sentences into clauses
Extracting noun chunks
Extracting entities and relations
Extracting subjects and objects of the sentence
Finding references – anaphora resolution
3
Chapter 3: Representing Text – Capturing Semantics
Chapter 3: Representing Text – Capturing Semantics
Technical requirements
Putting documents into a bag of words
Constructing the N-gram model
Representing texts with TF-IDF
Using word embeddings
Training your own embeddings model
Representing phrases – phrase2vec
Using BERT instead of word embeddings
Getting started with semantic search
4
Chapter 4: Classifying Texts
Chapter 4: Classifying Texts
Technical requirements
Getting the dataset and evaluation baseline ready
Performing rule-based text classification using keywords
Clustering sentences using K-means – unsupervised text classification
Using SVMs for supervised text classification
Using LSTMs for supervised text classification
5
Chapter 5: Getting Started with Information Extraction
Chapter 5: Getting Started with Information Extraction
Technical requirements
Using regular expressions
Finding similar strings: the Levenshtein distance
Performing named entity recognition using spaCy
Training your own NER model with spaCy
Discovering sentiment analysis
Sentiment for short texts using LSTM: Twitter
Using BERT for sentiment analysis
6
Chapter 6: Topic Modeling
Chapter 6: Topic Modeling
Technical requirements
LDA topic modeling with sklearn
LDA topic modeling with gensim
NMF topic modeling
K-means topic modeling with BERT
Topic modeling of short texts
7
Chapter 7: Building Chatbots
Chapter 7: Building Chatbots
Technical requirements
Building a basic chatbot with keyword matching
Building a basic Rasa chatbot
Creating question-answer pairs with Rasa
Creating and visualizing conversation paths with Rasa
Creating actions for the Rasa chatbot
8
Chapter 8: Visualizing Text Data
Chapter 8: Visualizing Text Data
Technical requirements
Visualizing the dependency parse
Visualizing parts of speech
Visualizing NER
Constructing word clouds
Visualizing topics
Why subscribe?
9
Other Books You May Enjoy
Other Books You May Enjoy
Packt is searching for authors like you
Leave a review - let other readers know what you think

Using SVMs for supervised text classification

In this recipe, we will build a machine learning classifier that uses the SVM algorithm. By the end of the recipe, you will have a working classifier that you will be able to test on new inputs and evaluate using the same classification_report tools as we used in the previous sections.

Getting ready

We will continue working with the same packages that we already installed in the previous recipes.

How to do it…

We will start with the already familiar steps of dividing data into training and testing sets and creating a vectorizer. We will then train the SVM classifier and evaluate it.

Your steps are as follows:

  1. Import the necessary functions and packages:
    import numpy as np
import pandas as pd
import string
import pickle
from sklearn import svm
from sklearn import preprocessing
from sklearn.metrics import classification_report
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction...
Previous Section
End of Section 6
Next Section