Analyzing XGBoost parameters
We will maintain the distinction between parameters and hyperparameters as presented in Chapter 2, Decision Trees in Depth. Hyperparameters are chosen before the model is trained, whereas parameters are chosen while the model is being trained. In other words, the parameters are what the model learns from the data.
The derivation that follows is taken from the XGBoost official documentation, Introduction to Boosted Trees, at https://xgboost.readthedocs.io/en/latest/tutorials/model.html.
The learning objective of a machine learning model determines how well the model fits the data. In the case of XGBoost, the learning objective consists of two parts: the loss function and the regularization term.
Mathematically, XGBoost's learning objective may be defined as follows:...