Book Image

Natural Language Processing with Python Quick Start Guide

By : Nirant Kasliwal
Book Image

Natural Language Processing with Python Quick Start Guide

By: Nirant Kasliwal

Overview of this book

NLP in Python is among the most sought after skills among data scientists. With code and relevant case studies, this book will show how you can use industry-grade tools to implement NLP programs capable of learning from relevant data. We will explore many modern methods ranging from spaCy to word vectors that have reinvented NLP. The book takes you from the basics of NLP to building text processing applications. We start with an introduction to the basic vocabulary along with a work?ow for building NLP applications. We use industry-grade NLP tools for cleaning and pre-processing text, automatic question and answer generation using linguistics, text embedding, text classifier, and building a chatbot. With each project, you will learn a new concept of NLP. You will learn about entity recognition, part of speech tagging and dependency parsing for Q and A. We use text embedding for both clustering documents and making chatbots, and then build classifiers using scikit-learn. We conclude by deploying these models as REST APIs with Flask. By the end, you will be confident building NLP applications, and know exactly what to look for when approaching new challenges.
Table of Contents (10 chapters)

Vectorizing a specific dataset

This section focuses almost exclusively on word vectors and how we can leverage the Gensim library to perform them.

Some of the questions we want to answer in this section include these:

  • How do we use original embedding, such as GLoVe?
  • How do we handle Out of Vocabulary words? (Hint: fastText)
  • How do we train our own word2vec vectors on our own corpus?
  • How do we train our own word2vec vectors?
  • How do we train our own fastText vectors?
  • How do we use similar words to compare both of the above?

First, let's get started with some simple imports, as follows:

import gensim
print(f'gensim: {gensim.__version__}')
> gensim: 3.4.0

Please ensure that your Gensim version is at least 3.4.0. This is a very popular package which is maintained and developed mostly by text processing experts over at RaRe Technologies. They use the same library in...