Besides the popular All-Reduce function, collective communication has a wide family of message-passing functions. We will discuss several of the most important collective communication functions in this section, namely Broadcast, Gather, and All-Gather.
Broadcast is also widely used in the All-Reduce architecture. For example, we can use Broadcast to distribute the initial model weights among all the workers. The following diagram shows an example of the Broadcast function in a three-workers setting:
As we can see, initially, Worker 1 holds a value of a. Worker 2 holds a value of b, while Worker 3 holds a value of c. Here, we conduct a broadcast from Worker 1 to all the other workers.