Book Image

Distributed Computing with Python

Book Image

Distributed Computing with Python

Overview of this book

CPU-intensive data processing tasks have become crucial considering the complexity of the various big data applications that are used today. Reducing the CPU utilization per process is very important to improve the overall speed of applications. This book will teach you how to perform parallel execution of computations by distributing them across multiple processors in a single machine, thus improving the overall performance of a big data processing task. We will cover synchronous and asynchronous models, shared memory and file systems, communication between various processes, synchronization, and more.
Table of Contents (15 chapters)
Distributed Computing with Python
Credits
About the Author
About the Reviewer
www.PacktPub.com
Preface
Index

Coroutines


In Python, the key to being able to suspend the execution of a function midway through is the use of coroutines, as we will see in this section. In order to understand coroutines, one needs to understand generators, and in order to understand those, one needs to have a grasp of iterators!

Most Python programmers are familiar with the concept of iterating some sort of collection (for example, strings, lists, tuples, file objects, and so on):

>>> for i in range(3):
...     print(i)
... 
0
1
2
>>> for line in open('exchange_rates_v1.py'):
...     print(line, end='')
... 
#!/usr/bin/env python3
import itertools
import time
import urllib.request

The reason why we can iterate all sorts of objects and not just lists or strings is the iteration protocol. The iteration protocol defines a standard interface for iteration: an object that implements __iter__ and __next__ (or __iter__ and next in Python 2.x) is an iterator and, as the name suggests, can be iterated over...