The cost function
We introduced the concept of the cost function in the Linear regression section in Chapter 4. The cost function gives us a mathematical way of determining how much error the current model has—it assigns a cost for making an incorrect prediction and provides a way to measure the model performance. The cost function is a key metric in ML model training—choosing the right cost function can improve model performance dramatically.
The common cost functions for regression models are MAE and MSE. As we have discussed in previous chapters, MAE defines a summation of the absolute differences between the prediction values and the label values. MSE defines the summation of squares of the differences between the prediction values and the label values.
The cost functions for classification models are quite different. Conceptually, the cost function for a classification model is the difference between the probability distributions for different classes. For...