Model monitoring and human-in-the-loop
In Chapter 11, we explored topics around bias detection, mitigation, and monitoring for large vision and language models. This was mostly in the context of evaluating your model. Now that we’ve made it to the section on deploying your models, with an extra focus on operations, let’s take a closer look at model monitoring.
Once you have a model deployed into any application, it’s extremely useful to be able to view the performance of that model over time. This is the case for any of the use cases we discussed earlier – chat, general search, forecasting, image generation, recommendations, classification, question answering, and so on. All of these applications benefit from being able to see how your model is trending over time and provide relevant alerts.
Imagine, for example, that you have a price forecasting model that suggests a price for a given product based on economic conditions. You train your model on certain...