Recurrent networks are essentially feedforward networks that retain state. All the networks we have looked at so far require an input of a fixed size, such as an image, and give a fixed size output, such as the probabilities of a particular class. Recurrent networks are different in that they accept a sequence, of arbitrary size, as the input and produce a sequence as output. Moreover, the internal state of the network's hidden layers is updated as a result of a learned function and the input. In this way, a recurrent network remembers its state. Subsequent states are a function of previous states.
In this chapter, we will cover the following:
- Introduction to recurrent networks
- Long short-term memory networks