So, we know how to create a model that is "adequate", but what does this really mean? How can we differentiate whether one "adequate" model is better than another? A common approach is to compare the ROC curves. This one is generated from the simple model that we just created:

You're probably familiar with ROC curves. They show us what kind of true positive rate we can achieve by allowing a given error rate in terms of false positives. The basic take away is that we want the curve to get as close to the upper left corner as possible. In case you haven't used these visualizations before, the reason for this is that the more the line is pulled up and to the left, the fewer false positives we get for every true positive. It maps very much to the concept of an error rate.

We have a visualization, which is great, but we can't automatically test it. We need to find some way to quantify this phenomenon. There is a simple, pretty straightforward way. It's called an Area Under...