Regularization
During the training process, the model is learning to find the best set of weights and biases that minimize the loss function. As the model architecture becomes more complex, or simply starts to take on more layers, the model is being fitted with more parameters. Although this may help to produce a better fit during training, having to use more parameters may also lead to overfitting.
In this section, we will dive into some regularization techniques that can be implemented in a straightforward fashion in the tf.keras API.
L1 and L2 regularization
Traditional methods to address the concern of overfitting involve introducing a penalty term in the loss function. This is known as regularization. The penalty term is directly related to model complexity, which is largely determined by the number of non-zero weights. To be more specific, there are three traditional types of regularization used in machine learning:
- L1 regularization (also known as Lasso):...