Chapter 3: Exploring Exhaustive Search
Hyperparameter tuning doesn't always correspond to fancy and complex search algorithms. In fact, a simple for
loop or manual search based on the developer's instinct can also be utilized to achieve the goal of hyperparameter tuning, which is to get the maximum evaluation score on the validation score without causing an overfitting issue.
In this chapter, we'll discuss the first out of four groups of hyperparameter tuning, called an exhaustive search. This is the most widely used and most straightforward hyperparameter-tuning group in practice. As explained by its name, hyperparameter-tuning methods that belong to this group work by exhaustively searching through the hyperparameter space. Except for one method, all of the methods in this group are categorized as uninformed search algorithms, meaning they are not learning from previous iterations to have a better search space in the future. Three methods will be discussed in this...