Chapter 6. Python on an HPC Cluster

In this chapter, we are going to look at yet another way of deploying our distributed Python applications. The method we are going to look at is the use of a High Performance Computing (HPC) cluster (also called a supercomputer), such as the multimillion dollar (or Euros), room-filling machines commonly found in evil overlord lairs.

Real HPC clusters are typically found in universities and national labs and are not what a typical start-up or small company would operate, mostly due to their cost. They are generally quite large systems with possibly hundreds of thousands of CPU cores spread over thousands of machines.

Oftentimes, the maximum size of HPC clusters a center can afford is determined by the amount of electricity that can be delivered to the premises; HPC systems that use a few megawatts are not at all uncommon. The system I work with, for instance, has over 160,000 cores across 7,000 nodes and consumes about 4 megawatts of electricity...