Model monitoring and drift detection
As was made clear in the previous section, in addition to error analysis, you should also make sure to monitor any potential shifts in your data. To do this, you could follow this process:
- Build and store a dataset with the training data of all the time series that you want to use to build a predictor.
- Compute the statistical characteristics of your dataset at a global level (for example, average, standard deviation, or histograms of the distribution of values).
- Compute the statistical characteristics of your dataset at a local level (for each time series).
- Train your predictors with these initial datasets and save the performance metrics (wQL, MAPE, and RMSE).
- Generate a new forecast based on this predictor; you will only get the predictions here and have no real data to compare them with yet.
- When new data comes in, compute the same statistical characteristics and compare them with the original values used at training...