12.2 MULTICOLLINEARITY
Data scientists need to guard against multicollinearity, a condition where some of the predictor variables are correlated with each other. Multicollinearity leads to instability in the solution space, leading, for example, to regression coefficients you cannot trust, because the coefficient variability is so large. Multicollinearity is an occupational hazard for data scientists, because many of the data sets have dozens if not hundreds of predictors, some of which are often correlated.
Consider Figures 12.1 and 12.2. Figure 12.1 illustrates a situation where the predictors x1 and x2 are not correlated with each other; that is, they are orthogonal, or independent. In such a case, the predictors form a solid basis, upon which the response surface y may rest sturdily, thereby providing stable coefficient estimates b1 and b2 each with small variability. On the other hand, Figure 12.2 illustrates a multicollinear situation where the predictors x1 and x2 are correlated...