Like sparse autoencoders, contractive autoencoders add a new regularization term to the loss function of the autoencoders. They try to make our encodings less sensitive to the small variations in the training data. So, with contractive autoencoders, our encodings become more robust to small perturbations such as noise present in our training dataset. We now introduce a new term called the regularizer or penalty term to our loss function. It helps to penalize the representations that are too sensitive to the input.
Our loss function can be mathematically represented as follows:
The first term represents the reconstruction error and the second term represents the penalty term or the regularizer and it is basically the Frobenius norm of the Jacobian matrix. Wait! What does that mean?
The Frobenius norm, also called the Hilbert-Schmidt norm...