In this section, we introduce an example using Keras. Keras is possibly the highest-level API for deep learning (again, at the time of writing, in this rapidly changing world of deep learning). This is very useful when you need to do production-ready models quite quickly, but is unfortunately sometimes not that great for learning, as everything is hidden away from you. Since, ideally, by the time you reach this section, an expert in recurrent neural networks, we can present you how to create a similar model.
Before that, let's introduce a simple benchmark model. Something that comes to mind when we speak about the memory of a neural network is the following, well, what if I had sufficient storage to calculate the conditional probabilities and simulate text generation as a Markov process, where the state variable is the observed text? We will implement this benchmark model to see how it compares in text generation quality with recurrent neural networks.