In this chapter, we explored the fundamental ideas surrounding predictive models. We saw that there are many ways to categorize models, learning important distinctions in the process, such as supervised versus unsupervised learning and regression versus classification. Next, we outlined the steps involved in building a predictive model, starting from the process of data collection all the way to model evaluation and deployment. Critically, this process is an iterative one, and most often we arrive at our final model after having tried out and trained a number of different models. In order to compare the performance of the different models that we create, we established some fundamental notions of model performance, such as the mean squared error for regression and the classification error rate for classification.

We also introduced our first model, the k-nearest neighbor model, which is useful in performing classification and regression alike. kNN is a very flexible model that doesn't make any explicit assumptions about the underlying data. Thus, it can fit a very complex decision boundary. It is a lazy learner, in that it doesn't construct a model to describe the relationship between the input features and the output variable. As a result, it doesn't require a long period of training. On the other hand, for data with many dimensions, it may take a long time to produce a prediction, and because the model needs to remember all the training data in order to find the nearest neighbors of a target point, it often also requires a lot of memory. kNN doesn't distinguish the importance of different features, and the fact that it uses a distance metric in its prediction means that, on the one hand, it does not have any built-in way to handle missing data and on the other, it often requires features to be transformed to similar scales. Finally, the model can be tuned by choosing an appropriate value of k, the number of nearest neighbors, so as to balance the degree of overfitting. With a firm grounding in the basics of the predictive modeling process, we will look at linear regression in the next chapter.