Book Image

Deep Learning for Natural Language Processing

By : Karthiek Reddy Bokka, Shubhangi Hora, Tanuj Jain, Monicah Wambugu
Book Image

Deep Learning for Natural Language Processing

By: Karthiek Reddy Bokka, Shubhangi Hora, Tanuj Jain, Monicah Wambugu

Overview of this book

Applying deep learning approaches to various NLP tasks can take your computational algorithms to a completely new level in terms of speed and accuracy. Deep Learning for Natural Language Processing starts by highlighting the basic building blocks of the natural language processing domain. The book goes on to introduce the problems that you can solve using state-of-the-art neural network models. After this, delving into the various neural network architectures and their specific areas of application will help you to understand how to select the best model to suit your needs. As you advance through this deep learning book, you’ll study convolutional, recurrent, and recursive neural networks, in addition to covering long short-term memory networks (LSTM). Understanding these networks will help you to implement their models using Keras. In later chapters, you will be able to develop a trigger word detection application using NLP techniques such as attention model and beam search. By the end of this book, you will not only have sound knowledge of natural language processing, but also be able to select the best text preprocessing and neural network models to solve a number of NLP issues.
Table of Contents (11 chapters)

Updates and Gradient Flow

The updates can be listed as follows:

  • Adjusting weight matrix Wy
  • Adjusting weight matrix Ws
  • For updating Wx

Adjusting Weight Matrix Wy

The model can be visualized as follows:

Figure 5.18: Back propagation of loss through weight matrix Wy

For Wy, the update is very simple since there are no additional paths or variables between Wy and the error. The matrix can be realized as follows:

Figure 5.19: Expression for weight matrix Wy

Adjusting Weight Matrix Ws

Figure 5.20: Back propagation of loss through weight matrix Ws with respect to S3

We can calculate the partial derivate of error with respect to Ws using the chain rule, as shown in the previous figure. It looks like that is what is needed, but it's important to remember that St is dependent on St-1, and therefore S3 is dependent on S2, so we need to consider S2 also, as shown here:

Figure 5.21: Back propagation of loss through weight matrix...