Despite CPython's GIL, it is possible to execute several tasks in parallel on multi-core computers using multiple processes instead of multiple threads. Python offers a native multiprocessing module. IPython offers an even simpler interface that brings powerful parallel computing features in an interactive environment. We will describe this tool here.
First, we launch four IPython engines in separate processes. We have basically two options to do this:
Executing
ipcluster start -n 4in a system shellUsing the web interface provided in the IPython notebook's main page by clicking on the Clusters tab and launching four engines
Then, we create a client that will act as a proxy to the IPython engines. The client automatically detects the running engines:
In [2]: from IPython.parallel import Client rc = Client()Let's check the number of running engines:
In [3]: rc.ids Out[3]: [0, 1, 2, 3]
To run commands in parallel over...