Book Image

Functional Python Programming - Second Edition

By : Steven F. Lott
Book Image

Functional Python Programming - Second Edition

By: Steven F. Lott

Overview of this book

If you’re a Python developer who wants to discover how to take the power of functional programming (FP) and bring it into your own programs, then this book is essential for you, even if you know next to nothing about the paradigm. Starting with a general overview of functional concepts, you’ll explore common functional features such as first-class and higher-order functions, pure functions, and more. You’ll see how these are accomplished in Python 3.6 to give you the core foundations you’ll build upon. After that, you’ll discover common functional optimizations for Python to help your apps reach even higher speeds. You’ll learn FP concepts such as lazy evaluation using Python’s generator functions and expressions. Moving forward, you’ll learn to design and implement decorators to create composite functions. You'll also explore data preparation techniques and data exploration in depth, and see how the Python standard library fits the functional programming model. Finally, to top off your journey into the world of functional Python, you’ll at look at the PyMonad project and some larger examples to put everything into perspective.
Table of Contents (22 chapters)
Title Page
Packt Upsell

Using a multiprocessing pool for concurrent processing

One elegant way to make use of the multiprocessing module is to create a processing Pool object and assign work to the various processes in that pool. We will use the OS to interleave execution among the various processes. If each of the processes has a mixture of I/O and computation, we should be able to ensure that our processor is very busy. When processes are waiting for the I/O to complete, other processes can do their computations. When an I/O finishes, a process will be ready to run and can compete with others for processing time.

The recipe for mapping work to a separate process looks like this:

import multiprocessing
    with multiprocessing.Pool(4) as workers:, glob.glob(pattern))  

We've created a Pool object with four separate processes and assigned this Pool object to the workers variable. We've then mapped a function, analysis, to an iterable queue of work to be done, using the pool of processes...