-
Book Overview & Buying
-
Table Of Contents
GPU-Accelerated Computing with Python 3 and CUDA
By :
In this chapter, we will explore how to use multiple GPUs when the limitations of a single GPU become a bottleneck. We will begin by highlighting the importance of multi-GPU computing, providing an overview of multi-GPU systems and two commonly used parallelism approaches. Next, we will present practical examples using Numba-CUDA to introduce core concepts such as data partitioning, memory data movement, and distributed computing. We will then scale these examples to a multi-node environment using a Dask cluster, which enables us to orchestrate computation across multiple GPUs and machines. Finally, we will demonstrate multi-GPU computing in JAX and how to perform distributed training of a machine learning model.
By the end of this chapter, you will have a solid understanding of the following: