Book Image

Functional Python Programming

By : Steven F. Lott, Steven F. Lott
Book Image

Functional Python Programming

By: Steven F. Lott, Steven F. Lott

Overview of this book

Table of Contents (23 chapters)
Functional Python Programming
Credits
About the Author
About the Reviewers
www.PacktPub.com
Preface
Index

Using a multiprocessing pool for concurrent processing


One elegant way to make use of the multiprocessing module is to create a processing Pool object and assign work to the various processes in that pool. We will use the OS to interleave execution among the various processes. If each of the processes has a mixture of I/O and computation, we should be able to assure that our processor is very busy. When processes are waiting for I/O to complete, other processes can do their computation. When an I/O completes, a process will be ready to run and can compete with others for processing time.

The recipe for mapping work to a separate process looks as follows:

    import multiprocessing
    with multiprocessing.Pool(4) as workers:
        workers.map(analysis, glob.glob(pattern))

We've created a Pool object with four separate processes and assigned this Pool object to the workers variable. We've then mapped a function, analysis, to an iterable queue of work to be done, using the pool of processes...