Another common parallelization strategy is to spread the workload of computational processes across multiple machines (physical or virtual). Where local parallelization is limited, ultimately, by the number of CPUs, or the number of cores, or the combination of both on a single machine, machine-level parallelization is limited by the number of machines that can be thrown at a problem. In this day and age, with immense reservoirs of virtual machines able to be made available in public clouds and private data centers, it's relatively easy to scale the number of available machines to match the computational needs of a problem.
The basic design for this kind of horizontally scalable solution is more complicated than the design for a local solution—it has to accomplish the same tasks, but separate the ability to do those tasks so that...