Reinforcing your backtesting strategy
In ML, backtesting is a technique used in forecasting to provide the learning process with two datasets, as follows:
- A training dataset on which the model will be trained
- A testing dataset on which we will evaluate the performance of the model on data that was not seen during the training phase
As a reminder, here are the different elements of backtesting in Amazon Forecast, as outlined in Chapter 4, Training a Predictor with AutoML:
When dealing with time series data, the split must mainly be done on the temporal axis (and, to a lesser extent, on the item population) to prevent any data leak from the past data to the future. This is paramount to make your model robust enough for when it will have to deal with actual production data.
By default, when you leave the default parameter as is (when selecting AutoML or when selecting an algorithm manually), Amazon...