Deploying and operating models

Once you have trained and optimized an ML model, it is ready for deployment. Many data science teams, in practice, stop here and move the model to production as a Docker image, often embedded in a REST API using Flask or similar frameworks. However, as you can imagine, this is not always the best solution depending on your use case requirements. An ML or data engineer's responsibility doesn't stop here.

The deployment and operation of an ML pipeline can be best seen when testing the model on live data in production. A test is done to collect insights and data to continuously improve the model. Hence, collecting model performance over time is an essential step to guaranteeing and improving the performance of the model.

In general, we differentiate two architectures for ML-scoring pipelines, which we will briefly discuss in this section:

• Batch scoring using pipelines
• Real-time scoring using a container-based web service

These architectures are discussed in increasing order of operational complexity, with offline scoring being the least complex and asynchronous scoring being the more complex system. The complexity arises from the number of components involved in operating such a pipeline at scale.

Finally, we will investigate an efficient way of collecting runtimes, latency, and other operational metrics, as well as model performance. It's also good practice to log all scored requests in order to analyze and improve an ML model over time.

Both architectures, as well as the monitoring solutions, will be discussed in more detail and implemented in Chapter 12, Deploying and operating machine learning models.

Batch scoring using pipelines

With offline scoring, or batch scoring, we are talking about an offline process where you evaluate an ML model against a batch of data. The result of this scoring technique is usually not time-critical and the data to be scored is usually larger than the model. This process is usually used when an ML model is scored within another batch process, such as a daily, hourly, or weekly task.

Here is what we expect as input and output data:

• Input: A location to find the input data
• Output: A response with all the scores

While the input and output format is quite intuitive, we still want to give a list of examples of when such an architecture is used. The reason for this is that you can decide the proper architecture for your use case when dealing with a similar ML task. Here are some practical examples:

• A recommendation engine of a streaming service generates new recommendations every week for the upcoming week.
• A classification algorithm of a mobile telecommunication operator computes a churn score for every customer once a month.

If the model was trained on a distributed system, it is very common to perform batch scoring on the same system that was used for training as the scoring task is identical to computing the score for the test set.

Real-time scoring using a container-based web service

The term online synchronous scoring, or real-time scoring, refers to a technique where we score an ML model and instantly need the resulting score. This is very common in stream processing, where single events are scored in real time. It's obvious that this task is highly time-critical and the execution is blocked until the resulting score is computed.

Here is what we expect as input and output data:

• Input: One or multiple observations
• Output: A response with a single score

The input and output configuration is also quite intuitive. Here are some practical examples of typical real-time scoring use cases:

• An object detection algorithm in a self-driving vehicle detects obstacles so it can control and steer the vehicle safely around the objects.
• An object detection algorithm detects faces in a camera image and focuses the camera.
• A classification algorithm decides whether the current product on the conveyor meets the quality standards.

Models for online synchronous scoring services are often deployed to the cloud as parallelized services in a distributed cluster with a load balancer in front of them. This way, the scoring cluster can be easily scaled up or down when higher throughput is required. If the latency requirements are restricted, these models are also deployed to edge devices, such as mobile phones or industrial computers, in order to avoid a round trip of the data to the nearest cloud region.

Tracking model performance, telemetry, and data skew

Tracking the proper metrics of a deployed ML model is essential. While popular metrics about include consumed CPU time, RAM, GPU memory, as well as latency, we also want to focus on the model's scoring performance. As we have already seen, most real- world data has a dependency on time and so many habits change over time. Operating an ML model in production means also continuously guaranteeing the quality and performance of the model.

In order to track the model's performance, you can use a standard application monitoring tool, such as Azure Application Insights or any other highly scalable key-value database. This is important to understand how your users are using your model and what your model is predicting in production.

Another important insight is tracking the data used for scoring the model. If we keep this data, we can compare it to the training data used for the deployed model and compute the data skew between the training data and the scoring data. By defining a threshold for maximum model skew, we can use this as a trigger to re-train the model once the skew is too big. We will see this in action in Chapter 12, Deploying and operating machine learning models.