Solving for an optimal training time
Time is an interesting construct in training large vision and language models. On the one hand, you might consider it a hyperparameter, simply the number of epochs. On the other hand, you might consider it a facet of your training data, its total number of tokens or images. You might also consider it a fixed input to your project, your total compute budget. Most research teams I work with use their intuition and good judgment to use a combination of all of these.
As we learned earlier in the book, the proposed scaling laws provide an interesting theoretical tool you can use to predict the performance of your model. Their original author, Kaplan et al. (9), actually suggested that optimal usage of a given compute budget should stop “significantly before convergence.” They proposed this because of their proposed insight into large language models being more “sample efficient” than smaller ones.
However, 2022 saw these...