Of all the machine learning algorithms we have considered thus far, none have considered data as a sequence. To take sequence data into account, we extend neural networks that store outputs from prior iterations. This type of neural network is called an RNN. Consider the fully connected network formulation:
Here, the weights are given by A multiplied by the input layer, x, and then run through an activation function, 
, which gives the output layer, y.
If we have a sequence of input data, 
, we can adapt the fully connected layer to take prior inputs into account, as follows:
On top of this recurrent iteration to get the next input, we want to get the probability distribution output, as follows:
Once we have a full sequence output, 
, we can consider the target as a number or category by just considering the last output. See the following diagram for how a general...