In the previous section, we used a very simple model: a threshold on one of the dimensions. Throughout this book, you will see many other types of models, and we're not even going to cover everything that is out there.
What makes up a classification model? We can break it up into three parts:
The structure of the model: In this, we use a threshold on a single feature.
The search procedure: In this, we try every possible combination of feature and threshold.
The loss function: Using the loss function, we decide which of the possibilities is less bad (because we can rarely talk about the perfect solution). We can use the training error or just define this point the other way around and say that we want the best accuracy. Traditionally, people want the loss function to be minimum.
We can play around with these parts to get different results. For example, we can attempt to build a threshold that achieves minimal training error, but we will only test three values...