KNN is a lazy learner. Also known as instance-based learners, lazy learners simply store the training dataset with little or no processing. In contrast to eager learners such as simple linear regression, KNN does not estimate the parameters of a model that generalizes the training data during a training phase. Lazy learning has advantages and disadvantages. Training an eager learner is often computationally costly, but prediction with the resulting model is often inexpensive. For simple linear regression, prediction consists only of multiplying the learned coefficient by the feature, and adding the learned intercept parameter. A lazy learner can predict almost immediately, but making predictions can be costly. In the simplest implementation of KNN, prediction requires calculating the distances between a test instance and all training instances.
In contrast to most of the other models that we will discuss, KNN is a non-parametric model. A parametric...