The goal of ensemble methods is to combine the predictions of several individual estimators built with a given learning algorithm in order to solve a shared problem. Typically, an ensemble consists of two major components:
- A set of models
- A set of decision rules that govern how the results of these models are combined into a single output
The idea behind ensemble methods has much to do with the wisdom of the crowd concept. Rather than the opinion of a single expert, we consider the collective opinion of a group of individuals. In the context of machine learning, these individuals would be classifiers or regressors. The idea is that if we just ask a large enough number of classifiers, one of them ought to get it right.
A consequence of this procedure is that we get a multitude of opinions about any given problem. So, how do we know which classifier...