Chapter 10: Model Tracking and Tuning in Azure Databricks
In the previous chapter, we learned how to create machine learning and deep learning models, as well as how to load datasets during distributed training in Azure Databricks. Finding the right machine learning algorithm to solve a problem using machine learning is one thing, but finding the best hyperparameters is another equally or more complex task. In this chapter, we will focus on model tuning, deployment, and control by using MLflow as a Model Repository. We will also use Hyperopt to search for the best set of hyperparameters for our models. We will implement the use of these libraries using deep learning models that have been made using the scikit-learn Python library.
More concretely, we will learn how to track runs of the machine learning model's training to find the most optimal set of hyperparameters, deploy and manage version control for the models using MLflow, and learn how to use Hyperopt as one of the...