You can now train a statistical language model from the prepared data.
The model that will be trained is a neural language model. It has a few unique characteristics:
- It uses a distributed representation for words so that different words with similar meanings will have a similar representation
- It learns the representation at the same time as learning the model
- It learns to predict the probability for the next word using the context of the previous 50 words
Specifically, you will use an Embedding Layer to learn the representation of words, and a Long Short-Term Memory (LSTM) recurrent neural network to learn to predict words based on their context.
The learned embedding needs to know the size of the vocabulary and the length of input sequences as previously discussed. It also has a parameter to specify how many dimensions will be used to represent each word. That is the size of the embedding vector space.
Common values are 50, 100, and 300. We will...