Checking the health of models with diagnostics
Throughout the previous chapters, we have primarily focused on the initial steps of predictive modeling, from data preparation and feature extraction to optimization of parameters. However, it is unlikely that our customers or business will remain unchanging, so predictive models must typically adapt as well. We can use a number of diagnostics to check the performance of models over time, which serve as a useful benchmark to evaluate the health of our algorithms.
Evaluating changes in model performance
Let us consider a scenario in which we train a predictive model on customer data and evaluate its performance on a set of new records each day for a month afterward. If this were a classification model, such as predicting whether a customer will cancel their subscription in the next pay period, we could use a metric such as the Area Under the Curve (AUC) of the Receiver-Operator-Characteristic (ROC) curve that we saw previously in Chapter 5, Putting...