We will start writing our own parallel executor. This executor will have no external dependencies, will be very light, and will be able to work on multiple core computers. We will also supply a version for clusters. There is no interprocess communication mechanism other than a shared filesystem. We will make a similar data analysis as in the previous recipe, but now in a real concurrent environment.
You should have read and understood the previous recipe. You will at least need to get the HapMap data and run the pickle part from it.
The code for this can be found in the 08_Advanced/Multiprocessing.ipynb notebook. Also, there is an external file called get_maf.py, which is available next to this notebook.