The initial chapters of this book provided you with some of the basic theories of parallel and distributed computing. They introduced a number of important concepts such as shared memory and distributed memory architectures and their differences.

They also looked at the basic arithmetic of code speedup by parallelization in terms of Amdahl's law. The main lesson of that discussion was that after a while, the efforts put into parallelizing an existing algorithm start to outweigh the performance gains. Also as mentioned, one way to side step Amdahl's law is to increase the problem size and have the parallel parts of our code do more work with respect to the serial parts (Gustafson's law).

The other lesson from Amdahl's law is to try and keep interprocess communication within our applications as small as possible. Ideally, our processes should be completely independent from each other. Little or no interprocess crosstalk reduces code complexity as well as general overhead...