Knowing when to stop and try again
The question "Is my model good enough?" is a whole subfield of machine learning on its own and can't fully be explained in a single chapter. Nevertheless, there's still some practical advice we all can follow to help us determine whether it's worth restarting training with different parameters, different model types, or even new data.
The two basic techniques for measuring success during and after training are as follows:
- Monitoring loss
- Assessing and comparing performance metrics
Monitoring loss
The first and most important thing you should look out for during training is whether the model is learning in an expected way. This is done by monitoring training output metrics such as loss. A typical training session follows the following pattern. In the initial stage of training, our weights are randomized, and loss will be huge (remember how bad our model with random weights was?). Then, even after one...