Section 3 – Advanced Parallelism Paradigms
In this section, we will learn state-of-the-art techniques on top of traditional data and model parallelism approaches. First, we will understand the concept of hybrid data-model parallelism. Second, we will discuss federated learning and edge device learning. Third, we will discuss elastic and in-parallel model training/inference in multitenant clusters or cloud environments. Finally, we will look at some more advanced techniques for further accelerating in-parallel model training and serving.
This section comprises the following chapters: