Book Image

Natural Language Processing Fundamentals

By : Sohom Ghosh, Dwight Gunning
Book Image

Natural Language Processing Fundamentals

By: Sohom Ghosh, Dwight Gunning

Overview of this book

If NLP hasn't been your forte, Natural Language Processing Fundamentals will make sure you set off to a steady start. This comprehensive guide will show you how to effectively use Python libraries and NLP concepts to solve various problems. You'll be introduced to natural language processing and its applications through examples and exercises. This will be followed by an introduction to the initial stages of solving a problem, which includes problem definition, getting text data, and preparing it for modeling. With exposure to concepts like advanced natural language processing algorithms and visualization techniques, you'll learn how to create applications that can extract information from unstructured data and present it as impactful visuals. Although you will continue to learn NLP-based techniques, the focus will gradually shift to developing useful applications. In these sections, you'll understand how to apply NLP techniques to answer questions as can be used in chatbots. By the end of this book, you'll be able to accomplish a varied range of assignments ranging from identifying the most suitable type of NLP task for solving a problem to using a tool like spacy or gensim for performing sentiment analysis. The book will easily equip you with the knowledge you need to build applications that interpret human language.
Table of Contents (10 chapters)

Generating Text with Markov Chains

One of the more interesting uses of NLP is generating text, whether for amusement, for research, or for profit. The key concept in generating text is to take advantage of the statistical properties of the text. This makes the output text more realistic and, more importantly, we gain information about the text itself if we can rearrange it to identify interesting patterns in it. Text can be generated in many different ways. We will explore doing so using Markov chains.

Markov Chains

A Markov chain is a mathematical system for transitioning between states based on probabilities. No matter how a system arrives at its present state, the possible future states are all predetermined. Furthermore, the move to the next state is based on a set of probabilities. Markov chains operate in timesteps. After each timestep, the system selects a new state. This new state is chosen based on the current state and the probabilities of the future states.

Markov...