Parameter server architecture
Workers are the computer nodes or GPUs that are responsible for model training. We split the total training data among all the workers. Each worker trains their local model with the training data partition that's been assigned to it.
The duties of parameter server are twofold:
- Aggregate model updates from all the workers.
- Update the model parameters held on the parameter server.
The following diagram depicts a simplified parameter server architecture with two workers and one...