To cement our understanding, let's start off by building the most basic autoencoder, as shown in the following diagram:
So far, we have emphasized that the hidden layer (Latent Representation) should be of a smaller dimension than the input data. This ensures that the latent representation is a compressed representation of the salient features of the input. But how small should it be?
Ideally, the size of the hidden layer should balance between being:
- Sufficiently small enough to represent a compressed representation of the input features
- Sufficiently large enough for the decoder to reconstruct the original input without too much loss
In other words, the size of the hidden layer is a hyperparameter that we need to select carefully to obtain the best results. We shall see how we can define the size of the hidden layer in Keras.
...