Book Image

Python Parallel Programming Cookbook

By : Giancarlo Zaccone
Book Image

Python Parallel Programming Cookbook

By: Giancarlo Zaccone

Overview of this book

This book will teach you parallel programming techniques using examples in Python and will help you explore the many ways in which you can write code that allows more than one process to happen at once. Starting with introducing you to the world of parallel computing, it moves on to cover the fundamentals in Python. This is followed by exploring the thread-based parallelism model using the Python threading module by synchronizing threads and using locks, mutex, semaphores queues, GIL, and the thread pool. Next you will be taught about process-based parallelism where you will synchronize processes using message passing along with learning about the performance of MPI Python Modules. You will then go on to learn the asynchronous parallel programming model using the Python asyncio module along with handling exceptions. Moving on, you will discover distributed computing with Python, and learn how to install a broker, use Celery Python Module, and create a worker. You will understand anche Pycsp, the Scoop framework, and disk modules in Python. Further on, you will learnGPU programming withPython using the PyCUDA module along with evaluating performance limitations.
Table of Contents (13 chapters)
Python Parallel Programming Cookbook
About the Author
About the Reviewers

Start working with processes in Python

On common operating systems, each program runs in its own process. Usually, we start a program by double-clicking on the icon's program or selecting it from a menu. In this recipe, we simply demonstrate how to start a single new program from inside a Python program. A process has its own space address, data stack, and other auxiliary data to keep track of the execution; the OS manages the execution of all processes, managing the access to the computational resources of the system via a scheduling procedure.

Getting ready

In this first Python application, you'll simply get the Python language installed.


Refer to to get the latest version of Python.

How to do it…

To execute this first example, we need to type the following two programs:



You can use the Python IDE (3.3.0) to edit these files:

The code for the file is as shown:

print ("Hello Python Parallel Cookbook!!")
closeInput = raw_input("Press ENTER to exit")
print "Closing calledProcess"

The code for the file is as shown:

##The following modules must be imported
import os
import sys

##this is the code to execute
program = "python"
print("Process calling")
arguments = [""]

##we call the script
os.execvp(program, (program,) + tuple(arguments))
print("Good Bye!!")

To run the example, open the program with the Python IDE and then press the F5 button on the keyboard.

You will see the following output in the Python shell:

At same time, the OS prompt displays the following:

We have two processes running to close the OS prompt; simply press the Enter button on the keyboard to do so.

How it works…

In the preceding example, the execvp function starts a new process, replacing the current one. Note that the "Good Bye" message is never printed. Instead, it searches for the program along the standard path, passes the contents of the second argument tuple as individual arguments to that program, and runs it with the current set of environment variables. The instruction input() in is only used to manage the closure of OS prompt. In the recipe dedicated to process-based parallelism, we will finally see how to manage a parallel execution of more processes via the multiprocessing Python module.