Ensembles of models (several models stacked together) can be conceptualized into two main groups: bagging and boosting. Bagging stands for bootstrap aggregation, meaning that several submodels are trained by bootstrapping (resampling with replacement) over the dataset. Each dataset will obviously be different and each model will yield different results. Boosting, on the other hand relies on training subsequent models using the residuals from the previous step. In each step, we have an aggregated model and a new model that is trained over those residuals. Both are combined to build a new combined model optimally (in such a way that the overall predictions are as good as possible).
The most famous bagging technique is random forests, which we have used previously in this chapter. Several boosting techniques have enjoyed an enormous popularity...