Model monitoring
Another important aspect of ML is model monitoring. There are different aspects of model monitoring: one could be system monitoring in the case of online models, where you monitor the latency, CPU, memory utilization, requests per minute of the model, and more. The other aspect is performance monitoring of the model. Again, there are many different ways of measuring performance. In this example, we will look at a simple classification report and the accuracy of the model.
To generate the classification report and calculate the accuracy of the live model, you need the prediction data and also the ground truth of the live data. For this example, let's say that the churn model is run once a week to generate the churn prediction and the ground truth will be available every 4 weeks from the day the model is run. That means if the model predicts customer x's churn as True
, and within the next 4 weeks, if we lose the customer for any reason, the model predicted...