So, let's get started with some statistical assessment work!
As we discussed in the previous section, instead of using all the data (the entire population of observations) to train a statistical model (and then test using some of that data), cross-validation divides the data into training and testing datasets.
The first step that a data scientist needs to take when he or she is interested in using cross-validation to assess the performance of a statistical model is to organize (or split) the data into two separate subsets.
There are actually several approaches of cross-validation:
- Leave-one-out cross-validation (LOOCV)
- Holdout
- k-fold and repeated k-fold
- Re-substitution (most agree that this method is the simplest method)
This cross-validation approaches all focus on how to split the data for the training, testing, and validation. Each has its own merit (pros and cons).
There are (as always) many approaches to programming a problem. The following is one such simple method...