Understanding hyperparameter space and distributions
Hyperparameter space is defined as the universal set of possible hyperparameter value combinations—in other words, it is the space containing all possible hyperparameter values that will be used as the search space during the hyperparameter-tuning phase. That's why it is also often called the hyperparameter-tuning search space. This space is predefined before the hyperparameter-tuning phase so that the search will be performed only on this space.
For example, we want to perform hyperparameter tuning on a NN. Let's say we want to search what is the best value for the dropout rate, the number of epochs, and batch-size hyperparameters.
The dropout rate is bounded in nature. Its value can only be between 0
and 1
, while for the number of epochs and batch-size hyperparameters, in theory, we can specify any positive integer value. However, there are other considerations that we need to think of. A higher...