Book Image

Hyperparameter Tuning with Python

By : Louis Owen
Book Image

Hyperparameter Tuning with Python

By: Louis Owen

Overview of this book

Hyperparameters are an important element in building useful machine learning models. This book curates numerous hyperparameter tuning methods for Python, one of the most popular coding languages for machine learning. Alongside in-depth explanations of how each method works, you will use a decision map that can help you identify the best tuning method for your requirements. You’ll start with an introduction to hyperparameter tuning and understand why it's important. Next, you'll learn the best methods for hyperparameter tuning for a variety of use cases and specific algorithm types. This book will not only cover the usual grid or random search but also other powerful underdog methods. Individual chapters are also dedicated to the three main groups of hyperparameter tuning methods: exhaustive search, heuristic search, Bayesian optimization, and multi-fidelity optimization. Later, you will learn about top frameworks like Scikit, Hyperopt, Optuna, NNI, and DEAP to implement hyperparameter tuning. Finally, you will cover hyperparameters of popular algorithms and best practices that will help you efficiently tune your hyperparameter. By the end of this book, you will have the skills you need to take full control over your machine learning models and get the best models for the best results.

Related Content you might be interested in

Current Title:

Small book image
Hyperparameter Tuning with Python
Louis Owen
Jul, 2022 | 306 pages
Small book image
Machine Learning with LightGBM and Python
Andrich van Wyk
Sep, 2023 | 252 pages
Small book image
Hands-On Genetic Algorithms with Python
Eyal Wirsansky
Jan, 2020 | 346 pages
Small book image
A Handbook of Mathematical Models with Python
Dr Ranja Sarkar
Aug, 2023 | 144 pages
Table of Contents (19 chapters)
Preface
Preface
Who this book is for
What this book covers
To get the most out of this book
Download the example code files
Download the color images
Conventions used
Get in touch
Share Your Thoughts
1
Section 1:The Methods
Section 1:The Methods
Free Chapter
2
Chapter 1: Evaluating Machine Learning Models
Chapter 1: Evaluating Machine Learning Models
Technical requirements
Understanding the concept of overfitting
Creating training, validation, and test sets
Exploring random and stratified splits
Discovering repeated k-fold cross-validation
Discovering Leave-One-Out cross-validation
Discovering LPO cross-validation
Discovering time-series cross-validation
Summary
Further reading
3
Chapter 2: Introducing Hyperparameter Tuning
Chapter 2: Introducing Hyperparameter Tuning
What is hyperparameter tuning?
Demystifying hyperparameters versus parameters
Understanding hyperparameter space 
and distributions
Summary
4
Chapter 3: Exploring Exhaustive Search
Chapter 3: Exploring Exhaustive Search
Understanding manual search
Understanding grid search
Understanding random search
Summary
5
Chapter 4: Exploring Bayesian Optimization
Chapter 4: Exploring Bayesian Optimization
Introducing BO
Understanding BO GP
Understanding SMAC
Understanding TPE
Understanding Metis
Summary
6
Chapter 5: Exploring Heuristic Search
Chapter 5: Exploring Heuristic Search
Understanding simulated annealing
Understanding genetic algorithms
Understanding particle swarm optimization
Understanding Population-Based Training
Summary
7
Chapter 6: Exploring Multi-Fidelity Optimization
Chapter 6: Exploring Multi-Fidelity Optimization
Introducing MFO
Understanding coarse-to-fine search
Understanding successive halving
Understanding hyper band
Understanding BOHB
Summary
8
Section 2:The Implementation
Section 2:The Implementation
9
Chapter 7: Hyperparameter Tuning via Scikit
Chapter 7: Hyperparameter Tuning via Scikit
Technical requirements
Introducing Scikit
Implementing Grid Search
Implementing Random Search
Implementing Coarse-to-Fine Search
Implementing Successive Halving
Implementing Hyper Band
Implementing Bayesian Optimization Gaussian Process
Implementing Bayesian Optimization Random Forest
Implementing Bayesian Optimization Gradient Boosted Trees
Summary
10
Chapter 8: Hyperparameter Tuning via Hyperopt
Chapter 8: Hyperparameter Tuning via Hyperopt
Technical requirements
Introducing Hyperopt
Implementing Random Search
Implementing Tree-structured Parzen Estimators
Implementing Adaptive TPE
Implementing simulated annealing
Summary
11
Chapter 9: Hyperparameter Tuning via Optuna
Chapter 9: Hyperparameter Tuning via Optuna
Technical requirements
Introducing Optuna
Implementing TPE
Implementing Random Search
Implementing Grid Search
Implementing Simulated Annealing
Implementing Successive Halving
Implementing Hyperband
Summary
12
Chapter 10: Advanced Hyperparameter Tuning with DEAP and Microsoft NNI
Chapter 10: Advanced Hyperparameter Tuning with DEAP and Microsoft NNI
Technical requirements
Introducing DEAP
Implementing the Genetic Algorithm
Implementing Particle Swarm Optimization
Introducing Microsoft NNI
Implementing Grid Search
Implementing Random Search
Implementing Tree-structured Parzen Estimators
Implementing Sequential Model Algorithm Configuration
Implementing Bayesian Optimization Gaussian Process
Implementing Metis
Implementing Simulated Annealing
Implementing Hyper Band
Implementing Bayesian Optimization Hyper Band
Implementing Population-Based Training
Summary
13
Section 3:Putting Things into Practice
Section 3:Putting Things into Practice
14
Chapter 11: Understanding the Hyperparameters of Popular Algorithms
Chapter 11: Understanding the Hyperparameters of Popular Algorithms
Exploring Random Forest hyperparameters
Exploring XGBoost hyperparameters
Exploring LightGBM hyperparameters
Exploring CatBoost hyperparameters
Exploring SVM hyperparameters
Exploring artificial neural network hyperparameters
Summary
15
Chapter 12: Introducing Hyperparameter Tuning Decision Map
Chapter 12: Introducing Hyperparameter Tuning Decision Map
Getting familiar with HTDM
Case study 1 – using HTDM with a CatBoost classifier
Case study 2 – using HTDM with a conditional hyperparameter space
Case study 3 – using HTDM with prior knowledge of the hyperparameter values
Summary
16
Chapter 13: Tracking Hyperparameter Tuning Experiments
Chapter 13: Tracking Hyperparameter Tuning Experiments
Technical requirements
Revisiting the usual practices
Exploring Neptune
Exploring scikit-optimize
Exploring Optuna
Exploring Microsoft NNI
Exploring MLflow
Summary
17
Chapter 14: Conclusions and Next Steps
Chapter 14: Conclusions and Next Steps
Revisiting hyperparameter tuning methods and packages
Revisiting HTDM
What’s next?
Summary
Why subscribe?
18
Other Books You May Enjoy
Other Books You May Enjoy
Packt is searching for authors like you
Share Your Thoughts

To get the most out of this book

You will also need Python version 3.7 (or above) installed on your computer, along with the related packages mentioned in the Technical requirements section of each chapter.

It is worth noting that there is a conflicting version requirement for the Hyperopt package in Chapter 8, Hyperparameter Tuning via Hyperopt, and Chapter 10, Advanced Hyperparameter Tuning with DEAP and Microsoft NNI. You need to install version 0.2.7 for Chapter 8, Hyperparameter Tuning via Hyperopt, and version 0.1.2 for Chapter 10, Advanced Hyperparameter Tuning with DEAP and Microsoft NNI.

It is also worth noting that the HyperBand implementation used in Chapter 7, Hyperparameter Tuning via Scikit, is the modified version of the scikit-hyperband package. You can utilize the modified version by cloning the GitHub repository (a link is available in the next section) and looking in a folder named hyperband.

If you are using the digital version of this book, we advise you to type the code yourself or access the code from the book’s GitHub repository (a link is available in the next section). Doing so will help you avoid any potential errors related to the copying and pasting of code.

To understand all contents in this book, you will need to have a basic understanding of ML and how to code in Python but will require no prior knowledge of hyperparameter tuning in Python. At the end of this book, you will also be introduced to several topics or implementations that you may benefit from which we have not covered yet in this book.

Previous Section
Next Section