Bringing it all home with examples from models today
Remember we learned earlier in the book that truly every state-of-the-art model requires some amount of distribution. This is because good models come from good datasets, and good datasets are large. These take time to process, so you need to distribute your processes in order to complete them in a timely manner. Some of them have models that are too big to fit on a single GPU, so they’ll require some amount of model parallelism. But others have models that are quite small, meaning they will only require data parallelism. Let’s step through two examples from top models today: Stable Diffusion and GPT-2.
Stable Diffusion – data parallelism at scale
Stable Diffusion is a fascinating model that enables you to create images from text. Once trained, you can simply provide textual input to Stable Diffusion, and it will generate a new picture for you! While researchers have been attempting this since at least...