-
Book Overview & Buying
-
Table Of Contents
Deep Learning with C++
By :
As you saw in the previous section in the case of GPT, the model sizes keep on growing, and this trend is expected to hold as the models become more capable with each iteration. The memory size of a GPU is also increasing with time, but the models’ growth has long surpassed the GPU memory growth. Now it is almost a standard practice to train models using multiple GPUs, either on a single node (computer) or using a multi-node setup (multiple computers connected within a network). Not only can you fit larger models, but you can also train the models faster. Here we will discuss two different and popular strategies for a multi-GPU training setup. We will look at C++ implementation for the same along with the theoretical concepts.
As transformer models grow and training datasets expand, single-GPU training becomes impractical. DDP training addresses this challenge by parallelizing the training process across multiple GPUs or machines, significantly...