Cross-validation, together with grid search, is commonly used to tune the hyperparameters of the model in order to achieve better performance. Below, we outline the differences between hyperparameters and parameters.
Hyperparameters:
- External characteristic of the model
- Not estimated based on data
- Can be considered the model's settings
- Set before the training phase
- Tuning them can result in better performance
Parameters:
- Internal characteristic of the model
- Estimated based on data, for example, the coefficients of linear regression
- Learned during the training phase
One of the challenges of machine learning is training models that are able to generalize well to unseen data (overfitting versus underfitting; a bias-variance trade-off). While tuning the model's hyperparameters, we would like to evaluate...