We've discussed how many of Python's built-ins and idioms that seem, at first blush, to be non-object-oriented are actually providing access to major objects under the hood. In this chapter, we'll discuss how the for loop that seems so structured is actually a lightweight wrapper around a set of object-oriented principles. We'll also see a variety of extensions to this syntax that automatically create even more types of object. We will cover:
- What design patterns are
- The iterator protocol—one of the most powerful design patterns
- List, set, and dictionary comprehensions
- Generators and coroutines
When engineers and architects decide to build a bridge, or a tower, or a building, they follow certain principles to ensure structural integrity. There are various possible designs for bridges (suspension or cantilever, for example), but if the engineer doesn't use one of the standard designs, and doesn't have a brilliant new design, it is likely the bridge he/she designs will collapse.
Design patterns are an attempt to bring this same formal definition for correctly designed structures to software engineering. There are many different design patterns to solve different general problems. People who create design patterns first identify a common problem faced by developers in a wide variety of situations. They then suggest what might be considered the ideal solution for that problem, in terms of object-oriented design.
Knowing a design pattern and choosing to use it in our software does not, however, guarantee that we are creating a "correct" solution. In 1907, the Québec Bridge (to this day, the longest cantilever bridge in the world) collapsed before construction was completed, because the engineers who designed it grossly underestimated the weight of the steel used to construct it. Similarly, in software development, we may incorrectly choose or apply a design pattern, and create software that "collapses" under normal operating situations or when stressed beyond its original design limits.
Any one design pattern proposes a set of objects interacting in a specific way to solve a general problem. The job of the programmer is to recognize when they are facing a specific version of that problem, and to adapt the general design in their solution.
In this chapter, we'll be covering the iterator design pattern. This pattern is so powerful and pervasive that the Python developers have provided multiple syntaxes to access the object-oriented principles underlying the pattern. We will be covering other design patterns in the next two chapters. Some of them have language support and some don't, but none of them is as intrinsically a part of the Python coder's daily life as the iterator pattern.
In typical design pattern parlance, an iterator is an object with a next() method and a done() method; the latter returns True if there are no items left in the sequence. In a programming language without built-in support for iterators, the iterator would be looped over like this:
while not iterator.done():
item = iterator.next()
# do something with the itemIn Python, iteration is a special feature, so the method gets a special name, __next__. This method can be accessed using the next(iterator) built-in. Rather than a done method, the iterator protocol raises StopIteration to notify the loop that it has completed. Finally, we have the much more readable for item in iterator syntax to actually access items in an iterator instead of messing around with a while loop. Let's look at these in more detail.
The abstract base class Iterator, in the collections.abc module, defines the iterator protocol in Python. As mentioned, it must have a __next__ method that the for loop (and other features that support iteration) can call to get a new element from the sequence. In addition, every iterator must also fulfill the Iterable interface. Any class that provides an __iter__ method is iterable; that method must return an Iterator instance that will cover all the elements in that class. Since an iterator is already looping over elements, its __iter__ function traditionally returns itself.
This might sound a bit confusing, so have a look at the following example, but note that this is a very verbose way to solve this problem. It clearly explains iteration and the two protocols in question, but we'll be looking at several more readable ways to get this effect later in this chapter:
class CapitalIterable:
def __init__(self, string):
self.string = string
def __iter__(self):
return CapitalIterator(self.string)
class CapitalIterator:
def __init__(self, string):
self.words = [w.capitalize() for w in string.split()]
self.index = 0
def __next__(self):
if self.index == len(self.words):
raise StopIteration()
word = self.words[self.index]
self.index += 1
return word
def __iter__(self):
return selfThis example defines an CapitalIterable class whose job is to loop over each of the words in a string and output them with the first letter capitalized. Most of the work of that iterable is passed to the CapitalIterator implementation. The canonical way to interact with this iterator is as follows:
>>> iterable = CapitalIterable('the quick brown fox jumps over the lazy dog') >>> iterator = iter(iterable) >>> while True: ... try: ... print(next(iterator)) ... except StopIteration: ... break ... The Quick Brown Fox Jumps Over The Lazy Dog
This example first constructs an iterable and retrieves an iterator from it. The distinction may need explanation; the iterable is an object with elements that can be looped over. Normally, these elements can be looped over multiple times, maybe even at the same time or in overlapping code. The iterator, on the other hand, represents a specific location in that iterable; some of the items have been consumed and some have not. Two different iterators might be at different places in the list of words, but any one iterator can mark only one place.
Each time next() is called on the iterator, it returns another token from the iterable, in order. Eventually, the iterator will be exhausted (won't have any more elements to return), in which case Stopiteration is raised, and we break out of the loop.
Of course, we already know a much simpler syntax for constructing an iterator from an iterable:
>>> for i in iterable: ... print(i) ... The Quick Brown Fox Jumps Over The Lazy Dog
As you can see, the for statement, in spite of not looking terribly object-oriented, is actually a shortcut to some obviously object-oriented design principles. Keep this in mind as we discuss comprehensions, as they, too, appear to be the polar opposite of an object-oriented tool. Yet, they use the exact same iteration protocol as for loops and are just another kind of shortcut.
The iterator protocol
The abstract base class Iterator, in the collections.abc module, defines the iterator protocol in Python. As mentioned, it must have a __next__ method that the for loop (and other features that support iteration) can call to get a new element from the sequence. In addition, every iterator must also fulfill the Iterable interface. Any class that provides an __iter__ method is iterable; that method must return an Iterator instance that will cover all the elements in that class. Since an iterator is already looping over elements, its __iter__ function traditionally returns itself.
This might sound a bit confusing, so have a look at the following example, but note that this is a very verbose way to solve this problem. It clearly explains iteration and the two protocols in question, but we'll be looking at several more readable ways to get this effect later in this chapter:
class CapitalIterable:
def __init__(self, string):
self.string = string
def __iter__(self):
return CapitalIterator(self.string)
class CapitalIterator:
def __init__(self, string):
self.words = [w.capitalize() for w in string.split()]
self.index = 0
def __next__(self):
if self.index == len(self.words):
raise StopIteration()
word = self.words[self.index]
self.index += 1
return word
def __iter__(self):
return selfThis example defines an CapitalIterable class whose job is to loop over each of the words in a string and output them with the first letter capitalized. Most of the work of that iterable is passed to the CapitalIterator implementation. The canonical way to interact with this iterator is as follows:
>>> iterable = CapitalIterable('the quick brown fox jumps over the lazy dog') >>> iterator = iter(iterable) >>> while True: ... try: ... print(next(iterator)) ... except StopIteration: ... break ... The Quick Brown Fox Jumps Over The Lazy Dog
This example first constructs an iterable and retrieves an iterator from it. The distinction may need explanation; the iterable is an object with elements that can be looped over. Normally, these elements can be looped over multiple times, maybe even at the same time or in overlapping code. The iterator, on the other hand, represents a specific location in that iterable; some of the items have been consumed and some have not. Two different iterators might be at different places in the list of words, but any one iterator can mark only one place.
Each time next() is called on the iterator, it returns another token from the iterable, in order. Eventually, the iterator will be exhausted (won't have any more elements to return), in which case Stopiteration is raised, and we break out of the loop.
Of course, we already know a much simpler syntax for constructing an iterator from an iterable:
>>> for i in iterable: ... print(i) ... The Quick Brown Fox Jumps Over The Lazy Dog
As you can see, the for statement, in spite of not looking terribly object-oriented, is actually a shortcut to some obviously object-oriented design principles. Keep this in mind as we discuss comprehensions, as they, too, appear to be the polar opposite of an object-oriented tool. Yet, they use the exact same iteration protocol as for loops and are just another kind of shortcut.
Comprehensions are simple, but powerful, syntaxes that allow us to transform or filter an iterable object in as little as one line of code. The resultant object can be a perfectly normal list, set, or dictionary, or it can be a generator expression that can be efficiently consumed in one go.
List comprehensions are one of the most powerful tools in Python, so people tend to think of them as advanced. They're not. Indeed, I've taken the liberty of littering previous examples with comprehensions and assuming you'd understand them. While it's true that advanced programmers use comprehensions a lot, it's not because they're advanced, it's because they're trivial, and handle some of the most common operations in software development.
Let's have a look at one of those common operations; namely, converting a list of items into a list of related items. Specifically, let's assume we just read a list of strings from a file, and now we want to convert it to a list of integers. We know every item in the list is an integer, and we want to do some activity (say, calculate an average) on those numbers. Here's one simple way to approach it:
input_strings = ['1', '5', '28', '131', '3']
output_integers = []
for num in input_strings:
output_integers.append(int(num))This works fine and it's only three lines of code. If you aren't used to comprehensions, you may not even think it looks ugly! Now, look at the same code using a list comprehension:
input_strings = ['1', '5', '28', '131', '3']output_integers = [int(num) for num in input_strings]
We're down to one line and, importantly for performance, we've dropped an append method call for each item in the list. Overall, it's pretty easy to tell what's going on, even if you're not used to comprehension syntax.
The square brackets indicate, as always, that we're creating a list. Inside this list is a for loop that iterates over each item in the input sequence. The only thing that may be confusing is what's happening between the list's opening brace and the start of the for loop. Whatever happens here is applied to each of the items in the input list. The item in question is referenced by the num variable from the loop. So, it's converting each individual element to an int data type.
That's all there is to a basic list comprehension. They are not so advanced after all. Comprehensions are highly optimized C code; list comprehensions are far faster than for loops when looping over a huge number of items. If readability alone isn't a convincing reason to use them as much as possible, speed should be.
Converting one list of items into a related list isn't the only thing we can do with a list comprehension. We can also choose to exclude certain values by adding an if statement inside the comprehension. Have a look:
output_ints = [int(n) for n in input_strings if len(n) < 3]
I shortened the name of the variable from num to n and the result variable to output_ints so it would still fit on one line. Other than this, all that's different between this example and the previous one is the if len(n) < 3 part. This extra code excludes any strings with more than two characters. The if statement is applied before the int function, so it's testing the length of a string. Since our input strings are all integers at heart, it excludes any number over 99. Now that is all there is to list comprehensions! We use them to map input values to output values, applying a filter along the way to include or exclude any values that meet a specific condition.
Any iterable can be the input to a list comprehension; anything we can wrap in a for loop can also be placed inside a comprehension. For example, text files are iterable; each call to __next__ on the file's iterator will return one line of the file. We could load a tab delimited file where the first line is a header row into a dictionary using the zip function:
import sys
filename = sys.argv[1]
with open(filename) as file:
header = file.readline().strip().split('\t')
contacts = [
dict(
zip(header, line.strip().split('\t'))
) for line in file
]
for contact in contacts:
print("email: {email} -- {last}, {first}".format(
**contact))This time, I've added some whitespace to make it somewhat more readable (list comprehensions don't have to fit on one line). This example creates a list of dictionaries from the zipped header and split lines for each line in the file.
Er, what? Don't worry if that code or explanation doesn't make sense; it's a bit confusing. One list comprehension is doing a pile of work here, and the code is hard to understand, read, and ultimately, maintain. This example shows that list comprehensions aren't always the best solution; most programmers would agree that a for loop would be more readable than this version.
Comprehensions aren't restricted to lists. We can use a similar syntax with braces to create sets and dictionaries as well. Let's start with sets. One way to create a set is to wrap a list comprehension in the set() constructor, which converts it to a set. But why waste memory on an intermediate list that gets discarded, when we can create a set directly?
Here's an example that uses a named tuple to model author/title/genre triads, and then retrieves a set of all the authors that write in a specific genre:
from collections import namedtuple
Book = namedtuple("Book", "author title genre")
books = [
Book("Pratchett", "Nightwatch", "fantasy"),
Book("Pratchett", "Thief Of Time", "fantasy"),
Book("Le Guin", "The Dispossessed", "scifi"),
Book("Le Guin", "A Wizard Of Earthsea", "fantasy"),
Book("Turner", "The Thief", "fantasy"),
Book("Phillips", "Preston Diamond", "western"),
Book("Phillips", "Twice Upon A Time", "scifi"),
]
fantasy_authors = {
b.author for b in books if b.genre == 'fantasy'}
The highlighted set comprehension sure is short in comparison to the demo-data setup! If we were to use a list comprehension, of course, Terry Pratchett would have been listed twice.. As it is, the nature of sets removes the duplicates, and we end up with:
>>> fantasy_authors {'Turner', 'Pratchett', 'Le Guin'}
We can introduce a colon to create a dictionary comprehension. This converts a sequence into a dictionary using key: value pairs. For example, it may be useful to quickly look up the author or genre in a dictionary if we know the title. We can use a dictionary comprehension to map titles to book objects:
fantasy_titles = {
b.title: b for b in books if b.genre == 'fantasy'}Now, we have a dictionary, and can look up books by title using the normal syntax.
In summary, comprehensions are not advanced Python, nor are they "non-object-oriented" tools that should be avoided. They are simply a more concise and optimized syntax for creating a list, set, or dictionary from an existing sequence.
Sometimes we want to process a new sequence without placing a new list, set, or dictionary into system memory. If we're just looping over items one at a time, and don't actually care about having a final container object created, creating that container is a waste of memory. When processing one item at a time, we only need the current object stored in memory at any one moment. But when we create a container, all the objects have to be stored in that container before we start processing them.
For example, consider a program that processes log files. A very simple log might contain information in this format:
Jan 26, 2015 11:25:25 DEBUG This is a debugging message. Jan 26, 2015 11:25:36 INFO This is an information method. Jan 26, 2015 11:25:46 WARNING This is a warning. It could be serious. Jan 26, 2015 11:25:52 WARNING Another warning sent. Jan 26, 2015 11:25:59 INFO Here's some information. Jan 26, 2015 11:26:13 DEBUG Debug messages are only useful if you want to figure something out. Jan 26, 2015 11:26:32 INFO Information is usually harmless, but helpful. Jan 26, 2015 11:26:40 WARNING Warnings should be heeded. Jan 26, 2015 11:26:54 WARNING Watch for warnings.
Log files for popular web servers, databases, or e-mail servers can contain many gigabytes of data (I recently had to clean nearly 2 terabytes of logs off a misbehaving system). If we want to process each line in the log, we can't use a list comprehension; it would create a list containing every line in the file. This probably wouldn't fit in RAM and could bring the computer to its knees, depending on the operating system.
If we used a for loop on the log file, we could process one line at a time before reading the next one into memory. Wouldn't be nice if we could use comprehension syntax to get the same effect?
This is where generator expressions come in. They use the same syntax as comprehensions, but they don't create a final container object. To create a generator expression, wrap the comprehension in () instead of [] or {}.
The following code parses a log file in the previously presented format, and outputs a new log file that contains only the WARNING lines:
import sys
inname = sys.argv[1]
outname = sys.argv[2]
with open(inname) as infile:
with open(outname, "w") as outfile:
warnings = (l for l in infile if 'WARNING' in l)
for l in warnings:
outfile.write(l)This program takes the two filenames on the command line, uses a generator expression to filter out the warnings (in this case, it uses the if syntax, and leaves the line unmodified), and then outputs the warnings to another file. If we run it on our sample file, the output looks like this:
Jan 26, 2015 11:25:46 WARNING This is a warning. It could be serious. Jan 26, 2015 11:25:52 WARNING Another warning sent. Jan 26, 2015 11:26:40 WARNING Warnings should be heeded. Jan 26, 2015 11:26:54 WARNING Watch for warnings.
Of course, with such a short input file, we could have safely used a list comprehension, but if the file is millions of lines long, the generator expression will have a huge impact on both memory and speed.
Generator expressions are frequently most useful inside function calls. For example, we can call sum, min, or max, on a generator expression instead of a list, since these functions process one object at a time. We're only interested in the result, not any intermediate container.
In general, a generator expression should be used whenever possible. If we don't actually need a list, set, or dictionary, but simply need to filter or convert items in a sequence, a generator expression will be most efficient. If we need to know the length of a list, or sort the result, remove duplicates, or create a dictionary, we'll have to use the comprehension syntax.
List comprehensions
List comprehensions are one of the most powerful tools in Python, so people tend to think of them as advanced. They're not. Indeed, I've taken the liberty of littering previous examples with comprehensions and assuming you'd understand them. While it's true that advanced programmers use comprehensions a lot, it's not because they're advanced, it's because they're trivial, and handle some of the most common operations in software development.
Let's have a look at one of those common operations; namely, converting a list of items into a list of related items. Specifically, let's assume we just read a list of strings from a file, and now we want to convert it to a list of integers. We know every item in the list is an integer, and we want to do some activity (say, calculate an average) on those numbers. Here's one simple way to approach it:
input_strings = ['1', '5', '28', '131', '3']
output_integers = []
for num in input_strings:
output_integers.append(int(num))This works fine and it's only three lines of code. If you aren't used to comprehensions, you may not even think it looks ugly! Now, look at the same code using a list comprehension:
input_strings = ['1', '5', '28', '131', '3']output_integers = [int(num) for num in input_strings]
We're down to one line and, importantly for performance, we've dropped an append method call for each item in the list. Overall, it's pretty easy to tell what's going on, even if you're not used to comprehension syntax.
The square brackets indicate, as always, that we're creating a list. Inside this list is a for loop that iterates over each item in the input sequence. The only thing that may be confusing is what's happening between the list's opening brace and the start of the for loop. Whatever happens here is applied to each of the items in the input list. The item in question is referenced by the num variable from the loop. So, it's converting each individual element to an int data type.
That's all there is to a basic list comprehension. They are not so advanced after all. Comprehensions are highly optimized C code; list comprehensions are far faster than for loops when looping over a huge number of items. If readability alone isn't a convincing reason to use them as much as possible, speed should be.
Converting one list of items into a related list isn't the only thing we can do with a list comprehension. We can also choose to exclude certain values by adding an if statement inside the comprehension. Have a look:
output_ints = [int(n) for n in input_strings if len(n) < 3]
I shortened the name of the variable from num to n and the result variable to output_ints so it would still fit on one line. Other than this, all that's different between this example and the previous one is the if len(n) < 3 part. This extra code excludes any strings with more than two characters. The if statement is applied before the int function, so it's testing the length of a string. Since our input strings are all integers at heart, it excludes any number over 99. Now that is all there is to list comprehensions! We use them to map input values to output values, applying a filter along the way to include or exclude any values that meet a specific condition.
Any iterable can be the input to a list comprehension; anything we can wrap in a for loop can also be placed inside a comprehension. For example, text files are iterable; each call to __next__ on the file's iterator will return one line of the file. We could load a tab delimited file where the first line is a header row into a dictionary using the zip function:
import sys
filename = sys.argv[1]
with open(filename) as file:
header = file.readline().strip().split('\t')
contacts = [
dict(
zip(header, line.strip().split('\t'))
) for line in file
]
for contact in contacts:
print("email: {email} -- {last}, {first}".format(
**contact))This time, I've added some whitespace to make it somewhat more readable (list comprehensions don't have to fit on one line). This example creates a list of dictionaries from the zipped header and split lines for each line in the file.
Er, what? Don't worry if that code or explanation doesn't make sense; it's a bit confusing. One list comprehension is doing a pile of work here, and the code is hard to understand, read, and ultimately, maintain. This example shows that list comprehensions aren't always the best solution; most programmers would agree that a for loop would be more readable than this version.
Comprehensions aren't restricted to lists. We can use a similar syntax with braces to create sets and dictionaries as well. Let's start with sets. One way to create a set is to wrap a list comprehension in the set() constructor, which converts it to a set. But why waste memory on an intermediate list that gets discarded, when we can create a set directly?
Here's an example that uses a named tuple to model author/title/genre triads, and then retrieves a set of all the authors that write in a specific genre:
from collections import namedtuple
Book = namedtuple("Book", "author title genre")
books = [
Book("Pratchett", "Nightwatch", "fantasy"),
Book("Pratchett", "Thief Of Time", "fantasy"),
Book("Le Guin", "The Dispossessed", "scifi"),
Book("Le Guin", "A Wizard Of Earthsea", "fantasy"),
Book("Turner", "The Thief", "fantasy"),
Book("Phillips", "Preston Diamond", "western"),
Book("Phillips", "Twice Upon A Time", "scifi"),
]
fantasy_authors = {
b.author for b in books if b.genre == 'fantasy'}
The highlighted set comprehension sure is short in comparison to the demo-data setup! If we were to use a list comprehension, of course, Terry Pratchett would have been listed twice.. As it is, the nature of sets removes the duplicates, and we end up with:
>>> fantasy_authors {'Turner', 'Pratchett', 'Le Guin'}
We can introduce a colon to create a dictionary comprehension. This converts a sequence into a dictionary using key: value pairs. For example, it may be useful to quickly look up the author or genre in a dictionary if we know the title. We can use a dictionary comprehension to map titles to book objects:
fantasy_titles = {
b.title: b for b in books if b.genre == 'fantasy'}Now, we have a dictionary, and can look up books by title using the normal syntax.
In summary, comprehensions are not advanced Python, nor are they "non-object-oriented" tools that should be avoided. They are simply a more concise and optimized syntax for creating a list, set, or dictionary from an existing sequence.
Sometimes we want to process a new sequence without placing a new list, set, or dictionary into system memory. If we're just looping over items one at a time, and don't actually care about having a final container object created, creating that container is a waste of memory. When processing one item at a time, we only need the current object stored in memory at any one moment. But when we create a container, all the objects have to be stored in that container before we start processing them.
For example, consider a program that processes log files. A very simple log might contain information in this format:
Jan 26, 2015 11:25:25 DEBUG This is a debugging message. Jan 26, 2015 11:25:36 INFO This is an information method. Jan 26, 2015 11:25:46 WARNING This is a warning. It could be serious. Jan 26, 2015 11:25:52 WARNING Another warning sent. Jan 26, 2015 11:25:59 INFO Here's some information. Jan 26, 2015 11:26:13 DEBUG Debug messages are only useful if you want to figure something out. Jan 26, 2015 11:26:32 INFO Information is usually harmless, but helpful. Jan 26, 2015 11:26:40 WARNING Warnings should be heeded. Jan 26, 2015 11:26:54 WARNING Watch for warnings.
Log files for popular web servers, databases, or e-mail servers can contain many gigabytes of data (I recently had to clean nearly 2 terabytes of logs off a misbehaving system). If we want to process each line in the log, we can't use a list comprehension; it would create a list containing every line in the file. This probably wouldn't fit in RAM and could bring the computer to its knees, depending on the operating system.
If we used a for loop on the log file, we could process one line at a time before reading the next one into memory. Wouldn't be nice if we could use comprehension syntax to get the same effect?
This is where generator expressions come in. They use the same syntax as comprehensions, but they don't create a final container object. To create a generator expression, wrap the comprehension in () instead of [] or {}.
The following code parses a log file in the previously presented format, and outputs a new log file that contains only the WARNING lines:
import sys
inname = sys.argv[1]
outname = sys.argv[2]
with open(inname) as infile:
with open(outname, "w") as outfile:
warnings = (l for l in infile if 'WARNING' in l)
for l in warnings:
outfile.write(l)This program takes the two filenames on the command line, uses a generator expression to filter out the warnings (in this case, it uses the if syntax, and leaves the line unmodified), and then outputs the warnings to another file. If we run it on our sample file, the output looks like this:
Jan 26, 2015 11:25:46 WARNING This is a warning. It could be serious. Jan 26, 2015 11:25:52 WARNING Another warning sent. Jan 26, 2015 11:26:40 WARNING Warnings should be heeded. Jan 26, 2015 11:26:54 WARNING Watch for warnings.
Of course, with such a short input file, we could have safely used a list comprehension, but if the file is millions of lines long, the generator expression will have a huge impact on both memory and speed.
Generator expressions are frequently most useful inside function calls. For example, we can call sum, min, or max, on a generator expression instead of a list, since these functions process one object at a time. We're only interested in the result, not any intermediate container.
In general, a generator expression should be used whenever possible. If we don't actually need a list, set, or dictionary, but simply need to filter or convert items in a sequence, a generator expression will be most efficient. If we need to know the length of a list, or sort the result, remove duplicates, or create a dictionary, we'll have to use the comprehension syntax.
Set and dictionary comprehensions
Comprehensions aren't restricted to lists. We can use a similar syntax with braces to create sets and dictionaries as well. Let's start with sets. One way to create a set is to wrap a list comprehension in the set() constructor, which converts it to a set. But why waste memory on an intermediate list that gets discarded, when we can create a set directly?
Here's an example that uses a named tuple to model author/title/genre triads, and then retrieves a set of all the authors that write in a specific genre:
from collections import namedtuple
Book = namedtuple("Book", "author title genre")
books = [
Book("Pratchett", "Nightwatch", "fantasy"),
Book("Pratchett", "Thief Of Time", "fantasy"),
Book("Le Guin", "The Dispossessed", "scifi"),
Book("Le Guin", "A Wizard Of Earthsea", "fantasy"),
Book("Turner", "The Thief", "fantasy"),
Book("Phillips", "Preston Diamond", "western"),
Book("Phillips", "Twice Upon A Time", "scifi"),
]
fantasy_authors = {
b.author for b in books if b.genre == 'fantasy'}
The highlighted set comprehension sure is short in comparison to the demo-data setup! If we were to use a list comprehension, of course, Terry Pratchett would have been listed twice.. As it is, the nature of sets removes the duplicates, and we end up with:
>>> fantasy_authors {'Turner', 'Pratchett', 'Le Guin'}
We can introduce a colon to create a dictionary comprehension. This converts a sequence into a dictionary using key: value pairs. For example, it may be useful to quickly look up the author or genre in a dictionary if we know the title. We can use a dictionary comprehension to map titles to book objects:
fantasy_titles = {
b.title: b for b in books if b.genre == 'fantasy'}Now, we have a dictionary, and can look up books by title using the normal syntax.
In summary, comprehensions are not advanced Python, nor are they "non-object-oriented" tools that should be avoided. They are simply a more concise and optimized syntax for creating a list, set, or dictionary from an existing sequence.
Sometimes we want to process a new sequence without placing a new list, set, or dictionary into system memory. If we're just looping over items one at a time, and don't actually care about having a final container object created, creating that container is a waste of memory. When processing one item at a time, we only need the current object stored in memory at any one moment. But when we create a container, all the objects have to be stored in that container before we start processing them.
For example, consider a program that processes log files. A very simple log might contain information in this format:
Jan 26, 2015 11:25:25 DEBUG This is a debugging message. Jan 26, 2015 11:25:36 INFO This is an information method. Jan 26, 2015 11:25:46 WARNING This is a warning. It could be serious. Jan 26, 2015 11:25:52 WARNING Another warning sent. Jan 26, 2015 11:25:59 INFO Here's some information. Jan 26, 2015 11:26:13 DEBUG Debug messages are only useful if you want to figure something out. Jan 26, 2015 11:26:32 INFO Information is usually harmless, but helpful. Jan 26, 2015 11:26:40 WARNING Warnings should be heeded. Jan 26, 2015 11:26:54 WARNING Watch for warnings.
Log files for popular web servers, databases, or e-mail servers can contain many gigabytes of data (I recently had to clean nearly 2 terabytes of logs off a misbehaving system). If we want to process each line in the log, we can't use a list comprehension; it would create a list containing every line in the file. This probably wouldn't fit in RAM and could bring the computer to its knees, depending on the operating system.
If we used a for loop on the log file, we could process one line at a time before reading the next one into memory. Wouldn't be nice if we could use comprehension syntax to get the same effect?
This is where generator expressions come in. They use the same syntax as comprehensions, but they don't create a final container object. To create a generator expression, wrap the comprehension in () instead of [] or {}.
The following code parses a log file in the previously presented format, and outputs a new log file that contains only the WARNING lines:
import sys
inname = sys.argv[1]
outname = sys.argv[2]
with open(inname) as infile:
with open(outname, "w") as outfile:
warnings = (l for l in infile if 'WARNING' in l)
for l in warnings:
outfile.write(l)This program takes the two filenames on the command line, uses a generator expression to filter out the warnings (in this case, it uses the if syntax, and leaves the line unmodified), and then outputs the warnings to another file. If we run it on our sample file, the output looks like this:
Jan 26, 2015 11:25:46 WARNING This is a warning. It could be serious. Jan 26, 2015 11:25:52 WARNING Another warning sent. Jan 26, 2015 11:26:40 WARNING Warnings should be heeded. Jan 26, 2015 11:26:54 WARNING Watch for warnings.
Of course, with such a short input file, we could have safely used a list comprehension, but if the file is millions of lines long, the generator expression will have a huge impact on both memory and speed.
Generator expressions are frequently most useful inside function calls. For example, we can call sum, min, or max, on a generator expression instead of a list, since these functions process one object at a time. We're only interested in the result, not any intermediate container.
In general, a generator expression should be used whenever possible. If we don't actually need a list, set, or dictionary, but simply need to filter or convert items in a sequence, a generator expression will be most efficient. If we need to know the length of a list, or sort the result, remove duplicates, or create a dictionary, we'll have to use the comprehension syntax.
Generator expressions
Sometimes we want to process a new sequence without placing a new list, set, or dictionary into system memory. If we're just looping over items one at a time, and don't actually care about having a final container object created, creating that container is a waste of memory. When processing one item at a time, we only need the current object stored in memory at any one moment. But when we create a container, all the objects have to be stored in that container before we start processing them.
For example, consider a program that processes log files. A very simple log might contain information in this format:
Jan 26, 2015 11:25:25 DEBUG This is a debugging message. Jan 26, 2015 11:25:36 INFO This is an information method. Jan 26, 2015 11:25:46 WARNING This is a warning. It could be serious. Jan 26, 2015 11:25:52 WARNING Another warning sent. Jan 26, 2015 11:25:59 INFO Here's some information. Jan 26, 2015 11:26:13 DEBUG Debug messages are only useful if you want to figure something out. Jan 26, 2015 11:26:32 INFO Information is usually harmless, but helpful. Jan 26, 2015 11:26:40 WARNING Warnings should be heeded. Jan 26, 2015 11:26:54 WARNING Watch for warnings.
Log files for popular web servers, databases, or e-mail servers can contain many gigabytes of data (I recently had to clean nearly 2 terabytes of logs off a misbehaving system). If we want to process each line in the log, we can't use a list comprehension; it would create a list containing every line in the file. This probably wouldn't fit in RAM and could bring the computer to its knees, depending on the operating system.
If we used a for loop on the log file, we could process one line at a time before reading the next one into memory. Wouldn't be nice if we could use comprehension syntax to get the same effect?
This is where generator expressions come in. They use the same syntax as comprehensions, but they don't create a final container object. To create a generator expression, wrap the comprehension in () instead of [] or {}.
The following code parses a log file in the previously presented format, and outputs a new log file that contains only the WARNING lines:
import sys
inname = sys.argv[1]
outname = sys.argv[2]
with open(inname) as infile:
with open(outname, "w") as outfile:
warnings = (l for l in infile if 'WARNING' in l)
for l in warnings:
outfile.write(l)This program takes the two filenames on the command line, uses a generator expression to filter out the warnings (in this case, it uses the if syntax, and leaves the line unmodified), and then outputs the warnings to another file. If we run it on our sample file, the output looks like this:
Jan 26, 2015 11:25:46 WARNING This is a warning. It could be serious. Jan 26, 2015 11:25:52 WARNING Another warning sent. Jan 26, 2015 11:26:40 WARNING Warnings should be heeded. Jan 26, 2015 11:26:54 WARNING Watch for warnings.
Of course, with such a short input file, we could have safely used a list comprehension, but if the file is millions of lines long, the generator expression will have a huge impact on both memory and speed.
Generator expressions are frequently most useful inside function calls. For example, we can call sum, min, or max, on a generator expression instead of a list, since these functions process one object at a time. We're only interested in the result, not any intermediate container.
In general, a generator expression should be used whenever possible. If we don't actually need a list, set, or dictionary, but simply need to filter or convert items in a sequence, a generator expression will be most efficient. If we need to know the length of a list, or sort the result, remove duplicates, or create a dictionary, we'll have to use the comprehension syntax.
Generator expressions are actually a sort of comprehension too; they compress the more advanced (this time it really is more advanced!) generator syntax into one line. The greater generator syntax looks even less object-oriented than anything we've seen, but we'll discover that once again, it is a simple syntax shortcut to create a kind of object.
Let's take the log file example a little further. If we want to delete the WARNING column from our output file (since it's redundant: this file contains only warnings), we have several options, at various levels of readability. We can do it with a generator expression:
import sys
inname, outname = sys.argv[1:3]
with open(inname) as infile:
with open(outname, "w") as outfile:
warnings = (l.replace('\tWARNING', '')
for l in infile if 'WARNING' in l)
for l in warnings:
outfile.write(l)That's perfectly readable, though I wouldn't want to make the expression much more complicated than that. We could also do it with a normal for loop:
import sys
inname, outname = sys.argv[1:3]
with open(inname) as infile:
with open(outname, "w") as outfile:
for l in infile:
if 'WARNING' in l:
outfile.write(l.replace('\tWARNING', ''))
That's maintainable, but so many levels of indent in so few lines is kind of ugly. More alarmingly, if we wanted to do something different with the lines, rather than just printing them out, we'd have to duplicate the looping and conditional code, too. Now let's consider a truly object-oriented solution, without any shortcuts:
import sys inname, outname = sys.argv[1:3] class WarningFilter: def __init__(self, insequence): self.insequence = insequence def __iter__(self): return self def __next__(self): l = self.insequence.readline() while l and 'WARNING' not in l: l = self.insequence.readline() if not l: raise StopIteration return l.replace('\tWARNING', '') with open(inname) as infile: with open(outname, "w") as outfile: filter = WarningFilter(infile) for l in filter: outfile.write(l)
No doubt about it: that is so ugly and difficult to read that you may not even be able to tell what's going on. We created an object that takes a file object as input, and provides a __next__ method like any iterator.
This __next__ method reads lines from the file, discarding them if they are not WARNING lines. When it encounters a WARNING line, it returns it. Then the for loop will call __next__ again to process the next WARNING line. When we run out of lines, we raise StopIteration to tell the loop we're finished iterating. It's pretty ugly compared to the other examples, but it's also powerful; now that we have a class in our hands, we can do whatever we want with it.
With that background behind us, we finally get to see generators in action. This next example does exactly the same thing as the previous one: it creates an object with a __next__ method that raises StopIteration when it's out of inputs:
import sys inname, outname = sys.argv[1:3] def warnings_filter(insequence): for l in insequence: if 'WARNING' in l: yield l.replace('\tWARNING', '') with open(inname) as infile: with open(outname, "w") as outfile: filter = warnings_filter(infile) for l in filter: outfile.write(l)
OK, that's pretty readable, maybe... at least it's short. But what on earth is going on here, it makes no sense whatsoever. And what is yield, anyway?
In fact, yield is the key to generators. When Python sees yield in a function, it takes that function and wraps it up in an object not unlike the one in our previous example. Think of the yield statement as similar to the return statement; it exits the function and returns a line. Unlike return, however, when the function is called again (via next()), it will start where it left off―on the line after the yield statement―instead of at the beginning of the function. In this example, there is no line "after" the yield statement, so it jumps to the next iteration of the for loop. Since the yield statement is inside an if statement, it only yields lines that contain WARNING.
While it looks like this is just a function looping over the lines, it is actually creating a special type of object, a generator object:
>>> print(warnings_filter([])) <generator object warnings_filter at 0xb728c6bc>
I passed an empty list into the function to act as an iterator. All the function does is create and return a generator object. That object has __iter__ and __next__ methods on it, just like the one we created in the previous example. Whenever __next__ is called, the generator runs the function until it finds a yield statement. It then returns the value from yield, and the next time __next__ is called, it picks up where it left off.
This use of generators isn't that advanced, but if you don't realize the function is creating an object, it can seem like magic. This example was quite simple, but you can get really powerful effects by making multiple calls to yield in a single function; the generator will simply pick up at the most recent yield and continue to the next one.
Often, when we build a generator function, we end up in a situation where we want to yield data from another iterable object, possibly a list comprehension or generator expression we constructed inside the generator, or perhaps some external items that were passed into the function. This has always been possible by looping over the iterable and individually yielding each item. However, in Python version 3.3, the Python developers introduced a new syntax to make this a little more elegant.
Let's adapt the generator example a bit so that instead of accepting a sequence of lines, it accepts a filename. This would normally be frowned upon as it ties the object to a particular paradigm. When possible we should operate on iterators as input; this way the same function could be used regardless of whether the log lines came from a file, memory, or a web-based log aggregator. So the following example is contrived for pedagogical reasons.
This version of the code illustrates that your generator can do some basic setup before yielding information from another iterable (in this case, a generator expression):
import sys
inname, outname = sys.argv[1:3]
def warnings_filter(infilename):
with open(infilename) as infile:
yield from (
l.replace('\tWARNING', '')
for l in infile
if 'WARNING' in l
)
filter = warnings_filter(inname)
with open(outname, "w") as outfile:
for l in filter:
outfile.write(l)This code combines the for loop from the previous example into a generator expression. Notice how I put the three clauses of the generator expression (the transformation, the loop, and the filter) on separate lines to make them more readable. Notice also that this transformation didn't help enough; the previous example with a for loop was more readable.
So let's consider an example that is more readable than its alternative. It can be useful to construct a generator that yields data from multiple other generators. The itertools.chain function, for example, yields data from iterables in sequence until they have all been exhausted. This can be implemented far too easily using the yield from syntax, so let's consider a classic computer science problem: walking a general tree.
A common implementation of the general tree data structure is a computer's filesystem. Let's model a few folders and files in a Unix filesystem so we can use yield from to walk them effectively:
class File:
def __init__(self, name):
self.name = name
class Folder(File):
def __init__(self, name):
super().__init__(name)
self.children = []
root = Folder('')
etc = Folder('etc')
root.children.append(etc)
etc.children.append(File('passwd'))
etc.children.append(File('groups'))
httpd = Folder('httpd')
etc.children.append(httpd)
httpd.children.append(File('http.conf'))
var = Folder('var')
root.children.append(var)
log = Folder('log')
var.children.append(log)
log.children.append(File('messages'))
log.children.append(File('kernel'))This setup code looks like a lot of work, but in a real filesystem, it would be even more involved. We'd have to read data from the hard drive and structure it into the tree. Once in memory, however, the code that outputs every file in the filesystem is quite elegant:
def walk(file):
if isinstance(file, Folder):
yield file.name + '/'
for f in file.children:
yield from walk(f)
else:
yield file.name
If this code encounters a directory, it recursively asks walk() to generate a list of all files subordinate to each of its children, and then yields all that data plus its own filename. In the simple case that it has encountered a normal file, it just yields that name.
As an aside, solving the preceding problem without using a generator is tricky enough that this problem is a common interview question. If you answer it as shown like this, be prepared for your interviewer to be both impressed and somewhat irritated that you answered it so easily. They will likely demand that you explain exactly what is going on. Of course, armed with the principles you've leaned in this chapter, you won't have any problem.
The yield from syntax is a useful shortcut when writing chained generators, but it is more commonly used for a different purpose: piping data through coroutines. We'll see many examples of this in Chapter 13, Concurrency, but for now, let's discover what a coroutine is.
Yield items from another iterable
Often, when we build a generator function, we end up in a situation where we want to yield data from another iterable object, possibly a list comprehension or generator expression we constructed inside the generator, or perhaps some external items that were passed into the function. This has always been possible by looping over the iterable and individually yielding each item. However, in Python version 3.3, the Python developers introduced a new syntax to make this a little more elegant.
Let's adapt the generator example a bit so that instead of accepting a sequence of lines, it accepts a filename. This would normally be frowned upon as it ties the object to a particular paradigm. When possible we should operate on iterators as input; this way the same function could be used regardless of whether the log lines came from a file, memory, or a web-based log aggregator. So the following example is contrived for pedagogical reasons.
This version of the code illustrates that your generator can do some basic setup before yielding information from another iterable (in this case, a generator expression):
import sys
inname, outname = sys.argv[1:3]
def warnings_filter(infilename):
with open(infilename) as infile:
yield from (
l.replace('\tWARNING', '')
for l in infile
if 'WARNING' in l
)
filter = warnings_filter(inname)
with open(outname, "w") as outfile:
for l in filter:
outfile.write(l)This code combines the for loop from the previous example into a generator expression. Notice how I put the three clauses of the generator expression (the transformation, the loop, and the filter) on separate lines to make them more readable. Notice also that this transformation didn't help enough; the previous example with a for loop was more readable.
So let's consider an example that is more readable than its alternative. It can be useful to construct a generator that yields data from multiple other generators. The itertools.chain function, for example, yields data from iterables in sequence until they have all been exhausted. This can be implemented far too easily using the yield from syntax, so let's consider a classic computer science problem: walking a general tree.
A common implementation of the general tree data structure is a computer's filesystem. Let's model a few folders and files in a Unix filesystem so we can use yield from to walk them effectively:
class File:
def __init__(self, name):
self.name = name
class Folder(File):
def __init__(self, name):
super().__init__(name)
self.children = []
root = Folder('')
etc = Folder('etc')
root.children.append(etc)
etc.children.append(File('passwd'))
etc.children.append(File('groups'))
httpd = Folder('httpd')
etc.children.append(httpd)
httpd.children.append(File('http.conf'))
var = Folder('var')
root.children.append(var)
log = Folder('log')
var.children.append(log)
log.children.append(File('messages'))
log.children.append(File('kernel'))This setup code looks like a lot of work, but in a real filesystem, it would be even more involved. We'd have to read data from the hard drive and structure it into the tree. Once in memory, however, the code that outputs every file in the filesystem is quite elegant:
def walk(file):
if isinstance(file, Folder):
yield file.name + '/'
for f in file.children:
yield from walk(f)
else:
yield file.name
If this code encounters a directory, it recursively asks walk() to generate a list of all files subordinate to each of its children, and then yields all that data plus its own filename. In the simple case that it has encountered a normal file, it just yields that name.
As an aside, solving the preceding problem without using a generator is tricky enough that this problem is a common interview question. If you answer it as shown like this, be prepared for your interviewer to be both impressed and somewhat irritated that you answered it so easily. They will likely demand that you explain exactly what is going on. Of course, armed with the principles you've leaned in this chapter, you won't have any problem.
The yield from syntax is a useful shortcut when writing chained generators, but it is more commonly used for a different purpose: piping data through coroutines. We'll see many examples of this in Chapter 13, Concurrency, but for now, let's discover what a coroutine is.
Coroutines are extremely powerful constructs that are often confused with generators. Many authors inappropriately describe coroutines as "generators with a bit of extra syntax." This is an easy mistake to make, as, way back in Python 2.5, when coroutines were introduced, they were presented as "we added a send method to the generator syntax." This is further complicated by the fact that when you create a coroutine in Python, the object returned is a generator. The difference is actually a lot more nuanced and will make more sense after you've seen a few examples.
Note
While coroutines in Python are currently tightly coupled to the generator syntax, they are only superficially related to the iterator protocol we have been discussing. The upcoming (as this is published) Python 3.5 release makes coroutines a truly standalone object and will provide a new syntax to work with them.
The other thing to bear in mind is that coroutines are pretty hard to understand. They are not used all that often in the wild, and you could likely skip this section and happily develop in Python for years without missing or even encountering them. There are a couple libraries that use coroutines extensively (mostly for concurrent or asynchronous programming), but they are normally written such that you can use coroutines without actually understanding how they work! So if you get lost in this section, don't despair.
But you won't get lost, having studied the following examples. Here's one of the simplest possible coroutines; it allows us to keep a running tally that can be increased by arbitrary values:
def tally():
score = 0
while True:
increment = yield score
score += incrementThis code looks like black magic that couldn't possibly work, so we'll see it working before going into a line-by-line description. This simple object could be used by a scoring application for a baseball team. Separate tallies could be kept for each team, and their score could be incremented by the number of runs accumulated at the end of every half-inning. Look at this interactive session:
>>> white_sox = tally() >>> blue_jays = tally() >>> next(white_sox) 0 >>> next(blue_jays) 0 >>> white_sox.send(3) 3 >>> blue_jays.send(2) 2 >>> white_sox.send(2) 5 >>> blue_jays.send(4) 6
First we construct two tally objects, one for each team. Yes, they look like functions, but as with the generator objects in the previous section, the fact that there is a yield statement inside the function tells Python to put a great deal of effort into turning the simple function into an object.
We then call next() on each of the coroutine objects. This does the same thing as calling next on any generator, which is to say, it executes each line of code until it encounters a yield statement, returns the value at that point, and then pauses until the next next() call.
So far, then, there's nothing new. But look back at the yield statement in our coroutine:
increment = yield score
Unlike with generators, this yield function looks like it's supposed to return a value and assign it to a variable. This is, in fact, exactly what's happening. The coroutine is still paused at the yield statement and waiting to be activated again by another call to next().
Or rather, as you see in the interactive session, a call to a method called send(). The send() method does exactly the same thing as next() except that in addition to advancing the generator to the next yield statement. It also allows you to pass in a value from outside the generator. This value is assigned to the left side of the yield statement.
The thing that is really confusing for many people is the order in which this happens:
yieldoccurs and the generator pausessend()occurs from outside the function and the generator wakes up- The value sent in is assigned to the left side of the
yieldstatement - The generator continues processing until it encounters another
yieldstatement
So, in this particular example, after we construct the coroutine and advance it to the yield statement with a call to next(), each successive call to send() passes a value into the coroutine, which adds this value to its score, goes back to the top of the while loop, and keeps processing until it hits the yield statement. The yield statement returns a value, and this value becomes the return value of the most recent call to send. Don't miss that: the send() method does not just submit a value to the generator, it also returns the value from the upcoming yield statement, just like next(). This is how we define the difference between a generator and a coroutine: a generator only produces values, while a coroutine can also consume them.
Note
The behavior and syntax of next(i), i.__next__(), and i.send(value) are rather unintuitive and frustrating. The first is a normal function, the second is a special method, and the last is a normal method. But all three do the same thing: advance the generator until it yields a value and pause. Further, the next() function and associated method can be replicated by calling i.send(None). There is value to having two different method names here, since it helps the reader of our code easily see whether they are interacting with a coroutine or a generator. I just find the fact that in one case it's a function call and in the other it's a normal method somewhat irritating.
Of course, the previous example could easily have been coded using a couple integer variables and calling x += increment on them. Let's look at a second example where coroutines actually save us some code. This example is a somewhat simplified (for pedagogical reasons) version of a problem I had to solve in my real job. The fact that it logically follows from the earlier discussions about processing a log file is completely serendipitous; those examples were written for the first edition of this book, whereas this problem came up four years later!
The Linux kernel log contains lines that look somewhat, but not quite completely, unlike this:
unrelated log messages sd 0:0:0:0 Attached Disk Drive unrelated log messages sd 0:0:0:0 (SERIAL=ZZ12345) unrelated log messages sd 0:0:0:0 [sda] Options unrelated log messages XFS ERROR [sda] unrelated log messages sd 2:0:0:1 Attached Disk Drive unrelated log messages sd 2:0:0:1 (SERIAL=ZZ67890) unrelated log messages sd 2:0:0:1 [sdb] Options unrelated log messages sd 3:0:1:8 Attached Disk Drive unrelated log messages sd 3:0:1:8 (SERIAL=WW11111) unrelated log messages sd 3:0:1:8 [sdc] Options unrelated log messages XFS ERROR [sdc] unrelated log messages
There are a whole bunch of interspersed kernel log messages, some of which pertain to hard disks. The hard disk messages might be interspersed with other messages, but they occur in a predictable format and order, in which a specific drive with a known serial number is associated with a bus identifier (such as 0:0:0:0), and a block device identifier (such as sda) is associated with that bus. Finally, if the drive has a corrupt filesystem, it might fail with an XFS error.
Now, given the preceding log file, the problem we need to solve is how to obtain the serial number of any drives that have XFS errors on them. This serial number might later be used by a data center technician to identify and replace the drive.
We know we can identify the individual lines using regular expressions, but we'll have to change the regular expressions as we loop through the lines, since we'll be looking for different things depending on what we found previously. The other difficult bit is that if we find an error string, the information about which bus contains that string, and what serial number is attached to the drive on that bus has already been processed. This can easily be solved by iterating through the lines of the file in reverse order.
Before you look at this example, be warned—the amount of code required for a coroutine-based solution is scarily small:
import re
def match_regex(filename, regex):
with open(filename) as file:
lines = file.readlines()
for line in reversed(lines):
match = re.match(regex, line)
if match:
regex = yield match.groups()[0]
def get_serials(filename):
ERROR_RE = 'XFS ERROR (\[sd[a-z]\])'
matcher = match_regex(filename, ERROR_RE)
device = next(matcher)
while True:
bus = matcher.send(
'(sd \S+) {}.*'.format(re.escape(device)))
serial = matcher.send('{} \(SERIAL=([^)]*)\)'.format(bus))
yield serial
device = matcher.send(ERROR_RE)
for serial_number in get_serials('EXAMPLE_LOG.log'):
print(serial_number)This code neatly divides the job into two separate tasks. The first task is to loop over all the lines and spit out any lines that match a given regular expression. The second task is to interact with the first task and give it guidance as to what regular expression it is supposed to be searching for at any given time.
Look at the match_regex coroutine first. Remember, it doesn't execute any code when it is constructed; rather, it just creates a coroutine object. Once constructed, someone outside the coroutine will eventually call next() to start the code running, at which point it stores the state of two variables, filename and regex. It then reads all the lines in the file and iterates over them in reverse. Each line is compared to the regular expression that was passed in until it finds a match. When the match is found, the coroutine yields the first group from the regular expression and waits.
At some point in the future, other code will send in a new regular expression to search for. Note that the coroutine never cares what regular expression it is trying to match; it's just looping over lines and comparing them to a regular expression. It's somebody else's responsibility to decide what regular expression to supply.
In this case, that somebody else is the get_serials generator. It doesn't care about the lines in the file, in fact it isn't even aware of them. The first thing it does is create a matcher object from the match_regex coroutine constructor, giving it a default regular expression to search for. It advances the coroutine to its first yield and stores the value it returns. It then goes into a loop that instructs the matcher object to search for a bus ID based on the stored device ID, and then a serial number based on that bus ID.
It idly yields that serial number to the outside for loop before instructing the matcher to find another device ID and repeat the cycle.
Basically, the coroutine's (match_regex, as it uses the regex = yield syntax) job is to search for the next important line in the file, while the generator's (get_serial, which uses the yield syntax without assignment) job is to decide which line is important. The generator has information about this particular problem, such as what order lines will appear in the file. The coroutine, on the other hand, could be plugged into any problem that required searching a file for given regular expressions.
Normal generators signal their exit from inside by raising StopIteration. If we chain multiple generators together (for example, by iterating over one generator from inside another), the StopIteration exception will be propagated outward. Eventually, it will hit a for loop that will see the exception and know that it's time to exit the loop.
Coroutines don't normally follow the iteration mechanism; rather than pulling data through one until an exception is encountered, data is usually pushed into it (using send). The entity doing the pushing is normally the one in charge of telling the coroutine when it's finished; it does this by calling the close() method on the coroutine in question.
When called, the close() method will raise a GeneratorExit exception at the point the coroutine was waiting for a value to be sent in. It is normally good policy for coroutines to wrap their yield statements in a try...finally block so that any cleanup tasks (such as closing associated files or sockets) can be performed.
If we need to raise an exception inside a coroutine, we can use the throw() method in a similar way. It accepts an exception type with optional value and traceback arguments. The latter is useful when we encounter an exception in one coroutine and want to cause an exception to occur in an adjacent coroutine while maintaining the traceback.
Both of these features are vital if you're building robust coroutine-based libraries, but we are unlikely to encounter them in our day-to-day coding lives.
We've seen coroutines in action, so now let's go back to that discussion of how they are related to generators. In Python, as is so often the case, the distinction is quite blurry. In fact, all coroutines are generator objects, and authors often use the two terms interchangeably. Sometimes, they describe coroutines as a subset of generators (only generators that return values from yield are considered coroutines). This is technically true in Python, as we've seen in the previous sections.
However, in the greater sphere of theoretical computer science, coroutines are considered the more general principles, and generators are a specific type of coroutine. Further, normal functions are yet another distinct subset of coroutines.
A coroutine is a routine that can have data passed in at one or more points and get it out at one or more points. In Python, the point where data is passed in and out is the yield statement.
A function, or subroutine, is the simplest type of coroutine. You can pass data in at one point, and get data out at one other point when the function returns. While a function can have multiple return statements, only one of them can be called for any given invocation of the function.
Finally, a generator is a type of coroutine that can have data passed in at one point, but can pass data out at multiple points. In Python, the data would be passed out at a yield statement, but you can't pass data back in. If you called send, the data would be silently discarded.
So in theory, generators are types of coroutines, functions are types of coroutines, and there are coroutines that are neither functions nor generators. That's simple enough, eh? So why does it feel more complicated in Python?
In Python, generators and coroutines are both constructed using a syntax that looks like we are constructing a function. But the resulting object is not a function at all; it's a totally different kind of object. Functions are, of course, also objects. But they have a different interface; functions are callable and return values, generators have data pulled out using next(), and coroutines have data pushed in using send.
Back to log parsing
Of course, the previous example could easily have been coded using a couple integer variables and calling x += increment on them. Let's look at a second example where coroutines actually save us some code. This example is a somewhat simplified (for pedagogical reasons) version of a problem I had to solve in my real job. The fact that it logically follows from the earlier discussions about processing a log file is completely serendipitous; those examples were written for the first edition of this book, whereas this problem came up four years later!
The Linux kernel log contains lines that look somewhat, but not quite completely, unlike this:
unrelated log messages sd 0:0:0:0 Attached Disk Drive unrelated log messages sd 0:0:0:0 (SERIAL=ZZ12345) unrelated log messages sd 0:0:0:0 [sda] Options unrelated log messages XFS ERROR [sda] unrelated log messages sd 2:0:0:1 Attached Disk Drive unrelated log messages sd 2:0:0:1 (SERIAL=ZZ67890) unrelated log messages sd 2:0:0:1 [sdb] Options unrelated log messages sd 3:0:1:8 Attached Disk Drive unrelated log messages sd 3:0:1:8 (SERIAL=WW11111) unrelated log messages sd 3:0:1:8 [sdc] Options unrelated log messages XFS ERROR [sdc] unrelated log messages
There are a whole bunch of interspersed kernel log messages, some of which pertain to hard disks. The hard disk messages might be interspersed with other messages, but they occur in a predictable format and order, in which a specific drive with a known serial number is associated with a bus identifier (such as 0:0:0:0), and a block device identifier (such as sda) is associated with that bus. Finally, if the drive has a corrupt filesystem, it might fail with an XFS error.
Now, given the preceding log file, the problem we need to solve is how to obtain the serial number of any drives that have XFS errors on them. This serial number might later be used by a data center technician to identify and replace the drive.
We know we can identify the individual lines using regular expressions, but we'll have to change the regular expressions as we loop through the lines, since we'll be looking for different things depending on what we found previously. The other difficult bit is that if we find an error string, the information about which bus contains that string, and what serial number is attached to the drive on that bus has already been processed. This can easily be solved by iterating through the lines of the file in reverse order.
Before you look at this example, be warned—the amount of code required for a coroutine-based solution is scarily small:
import re
def match_regex(filename, regex):
with open(filename) as file:
lines = file.readlines()
for line in reversed(lines):
match = re.match(regex, line)
if match:
regex = yield match.groups()[0]
def get_serials(filename):
ERROR_RE = 'XFS ERROR (\[sd[a-z]\])'
matcher = match_regex(filename, ERROR_RE)
device = next(matcher)
while True:
bus = matcher.send(
'(sd \S+) {}.*'.format(re.escape(device)))
serial = matcher.send('{} \(SERIAL=([^)]*)\)'.format(bus))
yield serial
device = matcher.send(ERROR_RE)
for serial_number in get_serials('EXAMPLE_LOG.log'):
print(serial_number)This code neatly divides the job into two separate tasks. The first task is to loop over all the lines and spit out any lines that match a given regular expression. The second task is to interact with the first task and give it guidance as to what regular expression it is supposed to be searching for at any given time.
Look at the match_regex coroutine first. Remember, it doesn't execute any code when it is constructed; rather, it just creates a coroutine object. Once constructed, someone outside the coroutine will eventually call next() to start the code running, at which point it stores the state of two variables, filename and regex. It then reads all the lines in the file and iterates over them in reverse. Each line is compared to the regular expression that was passed in until it finds a match. When the match is found, the coroutine yields the first group from the regular expression and waits.
At some point in the future, other code will send in a new regular expression to search for. Note that the coroutine never cares what regular expression it is trying to match; it's just looping over lines and comparing them to a regular expression. It's somebody else's responsibility to decide what regular expression to supply.
In this case, that somebody else is the get_serials generator. It doesn't care about the lines in the file, in fact it isn't even aware of them. The first thing it does is create a matcher object from the match_regex coroutine constructor, giving it a default regular expression to search for. It advances the coroutine to its first yield and stores the value it returns. It then goes into a loop that instructs the matcher object to search for a bus ID based on the stored device ID, and then a serial number based on that bus ID.
It idly yields that serial number to the outside for loop before instructing the matcher to find another device ID and repeat the cycle.
Basically, the coroutine's (match_regex, as it uses the regex = yield syntax) job is to search for the next important line in the file, while the generator's (get_serial, which uses the yield syntax without assignment) job is to decide which line is important. The generator has information about this particular problem, such as what order lines will appear in the file. The coroutine, on the other hand, could be plugged into any problem that required searching a file for given regular expressions.
Normal generators signal their exit from inside by raising StopIteration. If we chain multiple generators together (for example, by iterating over one generator from inside another), the StopIteration exception will be propagated outward. Eventually, it will hit a for loop that will see the exception and know that it's time to exit the loop.
Coroutines don't normally follow the iteration mechanism; rather than pulling data through one until an exception is encountered, data is usually pushed into it (using send). The entity doing the pushing is normally the one in charge of telling the coroutine when it's finished; it does this by calling the close() method on the coroutine in question.
When called, the close() method will raise a GeneratorExit exception at the point the coroutine was waiting for a value to be sent in. It is normally good policy for coroutines to wrap their yield statements in a try...finally block so that any cleanup tasks (such as closing associated files or sockets) can be performed.
If we need to raise an exception inside a coroutine, we can use the throw() method in a similar way. It accepts an exception type with optional value and traceback arguments. The latter is useful when we encounter an exception in one coroutine and want to cause an exception to occur in an adjacent coroutine while maintaining the traceback.
Both of these features are vital if you're building robust coroutine-based libraries, but we are unlikely to encounter them in our day-to-day coding lives.
We've seen coroutines in action, so now let's go back to that discussion of how they are related to generators. In Python, as is so often the case, the distinction is quite blurry. In fact, all coroutines are generator objects, and authors often use the two terms interchangeably. Sometimes, they describe coroutines as a subset of generators (only generators that return values from yield are considered coroutines). This is technically true in Python, as we've seen in the previous sections.
However, in the greater sphere of theoretical computer science, coroutines are considered the more general principles, and generators are a specific type of coroutine. Further, normal functions are yet another distinct subset of coroutines.
A coroutine is a routine that can have data passed in at one or more points and get it out at one or more points. In Python, the point where data is passed in and out is the yield statement.
A function, or subroutine, is the simplest type of coroutine. You can pass data in at one point, and get data out at one other point when the function returns. While a function can have multiple return statements, only one of them can be called for any given invocation of the function.
Finally, a generator is a type of coroutine that can have data passed in at one point, but can pass data out at multiple points. In Python, the data would be passed out at a yield statement, but you can't pass data back in. If you called send, the data would be silently discarded.
So in theory, generators are types of coroutines, functions are types of coroutines, and there are coroutines that are neither functions nor generators. That's simple enough, eh? So why does it feel more complicated in Python?
In Python, generators and coroutines are both constructed using a syntax that looks like we are constructing a function. But the resulting object is not a function at all; it's a totally different kind of object. Functions are, of course, also objects. But they have a different interface; functions are callable and return values, generators have data pulled out using next(), and coroutines have data pushed in using send.
Closing coroutines and throwing exceptions
Normal generators signal their exit from inside by raising StopIteration. If we chain multiple generators together (for example, by iterating over one generator from inside another), the StopIteration exception will be propagated outward. Eventually, it will hit a for loop that will see the exception and know that it's time to exit the loop.
Coroutines don't normally follow the iteration mechanism; rather than pulling data through one until an exception is encountered, data is usually pushed into it (using send). The entity doing the pushing is normally the one in charge of telling the coroutine when it's finished; it does this by calling the close() method on the coroutine in question.
When called, the close() method will raise a GeneratorExit exception at the point the coroutine was waiting for a value to be sent in. It is normally good policy for coroutines to wrap their yield statements in a try...finally block so that any cleanup tasks (such as closing associated files or sockets) can be performed.
If we need to raise an exception inside a coroutine, we can use the throw() method in a similar way. It accepts an exception type with optional value and traceback arguments. The latter is useful when we encounter an exception in one coroutine and want to cause an exception to occur in an adjacent coroutine while maintaining the traceback.
Both of these features are vital if you're building robust coroutine-based libraries, but we are unlikely to encounter them in our day-to-day coding lives.
We've seen coroutines in action, so now let's go back to that discussion of how they are related to generators. In Python, as is so often the case, the distinction is quite blurry. In fact, all coroutines are generator objects, and authors often use the two terms interchangeably. Sometimes, they describe coroutines as a subset of generators (only generators that return values from yield are considered coroutines). This is technically true in Python, as we've seen in the previous sections.
However, in the greater sphere of theoretical computer science, coroutines are considered the more general principles, and generators are a specific type of coroutine. Further, normal functions are yet another distinct subset of coroutines.
A coroutine is a routine that can have data passed in at one or more points and get it out at one or more points. In Python, the point where data is passed in and out is the yield statement.
A function, or subroutine, is the simplest type of coroutine. You can pass data in at one point, and get data out at one other point when the function returns. While a function can have multiple return statements, only one of them can be called for any given invocation of the function.
Finally, a generator is a type of coroutine that can have data passed in at one point, but can pass data out at multiple points. In Python, the data would be passed out at a yield statement, but you can't pass data back in. If you called send, the data would be silently discarded.
So in theory, generators are types of coroutines, functions are types of coroutines, and there are coroutines that are neither functions nor generators. That's simple enough, eh? So why does it feel more complicated in Python?
In Python, generators and coroutines are both constructed using a syntax that looks like we are constructing a function. But the resulting object is not a function at all; it's a totally different kind of object. Functions are, of course, also objects. But they have a different interface; functions are callable and return values, generators have data pulled out using next(), and coroutines have data pushed in using send.
The relationship between coroutines, generators, and functions
We've seen coroutines in action, so now let's go back to that discussion of how they are related to generators. In Python, as is so often the case, the distinction is quite blurry. In fact, all coroutines are generator objects, and authors often use the two terms interchangeably. Sometimes, they describe coroutines as a subset of generators (only generators that return values from yield are considered coroutines). This is technically true in Python, as we've seen in the previous sections.
However, in the greater sphere of theoretical computer science, coroutines are considered the more general principles, and generators are a specific type of coroutine. Further, normal functions are yet another distinct subset of coroutines.
A coroutine is a routine that can have data passed in at one or more points and get it out at one or more points. In Python, the point where data is passed in and out is the yield statement.
A function, or subroutine, is the simplest type of coroutine. You can pass data in at one point, and get data out at one other point when the function returns. While a function can have multiple return statements, only one of them can be called for any given invocation of the function.
Finally, a generator is a type of coroutine that can have data passed in at one point, but can pass data out at multiple points. In Python, the data would be passed out at a yield statement, but you can't pass data back in. If you called send, the data would be silently discarded.
So in theory, generators are types of coroutines, functions are types of coroutines, and there are coroutines that are neither functions nor generators. That's simple enough, eh? So why does it feel more complicated in Python?
In Python, generators and coroutines are both constructed using a syntax that looks like we are constructing a function. But the resulting object is not a function at all; it's a totally different kind of object. Functions are, of course, also objects. But they have a different interface; functions are callable and return values, generators have data pulled out using next(), and coroutines have data pushed in using send.
One of the fields in which Python is the most popular these days is data science. Let's implement a basic machine learning algorithm! Machine learning is a huge topic, but the general idea is to make predictions or classifications about future data by using knowledge gained from past data. Uses of such algorithms abound, and data scientists are finding new ways to apply machine learning every day. Some important machine learning applications include computer vision (such as image classification or facial recognition), product recommendation, identifying spam, and speech recognition. We'll look at a simpler problem: given an RGB color definition, what name would humans identify that color as?
There are more than 16 million colors in the standard RGB color space, and humans have come up with names for only a fraction of them. While there are thousands of names (some quite ridiculous; just go to any car dealership or makeup store), let's build a classifier that attempts to divide the RGB space into the basic colors:
- Red
- Purple
- Blue
- Green
- Yellow
- Orange
- Grey
- White
- Pink
The first thing we need is a dataset to train our algorithm on. In a production system, you might scrape a list of colors website or survey thousands of people. Instead, I created a simple application that renders a random color and asks the user to select one of the preceding nine options to classify it. This application is included with the example code for this chapter in the kivy_color_classifier directory, but we won't be going into the details of this code as its only purpose here is to generate sample data.
For the purposes of this case study, the important thing about that application is the output, which is a comma-separated value (CSV) file that contains four values per row: the red, green, and blue values (represented as a floating-point number between zero and one), and one of the preceding nine names that the user assigned to that color. The dataset looks something like this:
0.30928279150905513,0.7536768153744394,0.3244011790604804,Green 0.4991001855115986,0.6394567277907686,0.6340502030888825,Grey 0.21132621004927998,0.3307376167520666,0.704037576789711,Blue 0.7260420945787928,0.4025279573860123,0.49781705131696363,Pink 0.706469868610228,0.28530423638868196,0.7880240251003464,Purple 0.692243900051664,0.7053550777777416,0.1845069151913028,Yellow 0.3628979381122397,0.11079495501215897,0.26924540840045075,Purple 0.611273677646518,0.48798521783547677,0.5346130557761224,Purple . . . 0.4014121109376566,0.42176706818252674,0.9601866228083298,Blue 0.17750449496124632,0.8008214961070862,0.5073944321437429,Green
I made 200 datapoints (a very few of them untrue) before I got bored and decided it was time to start machine learning on this dataset. These datapoints are shipped with the examples for this chapter if you would like to use my data (nobody's ever told me I'm color-blind, so it should be somewhat reasonable).
We'll be implementing one of the simpler machine-learning algorithms, referred to as k-nearest neighbor. This algorithm relies on some kind of "distance" calculation between points in the dataset (in our case, we can use a three-dimensional version of the Pythagorean theorem). Given a new datapoint, it finds a certain number (referred to as k, as in k-nearest neighbors) of datapoints that are closest to it when measured by that distance calculation. Then it combines those datapoints in some way (an average might work for linear calculations; for our classification problem, we'll use the mode), and returns the result.
We won't go into too much detail about what the algorithm does; rather, we'll focus on some of the ways we can apply the iterator pattern or iterator protocol to this problem.
Let's now write a program that performs the following steps in order:
- Load the sample data from the file and construct a model from it.
- Generate 100 random colors.
- Classify each color and output it to a file in the same format as the input.
Once we have this second CSV file, another Kivy program can load the file and render each color, asking a human user to confirm or deny the accuracy of the prediction, thus informing us of how accurate our algorithm and initial data set are.
The first step is a fairly simple generator that loads CSV data and converts it into a format that is amenable to our needs:
import csv
dataset_filename = 'colors.csv'
def load_colors(filename):
with open(filename) as dataset_file:
lines = csv.reader(dataset_file)
for line in lines:
yield tuple(float(y) for y in line[0:3]), line[3]
We haven't seen the csv.reader function before. It returns an iterator over the lines in the file. Each value returned by the iterator is a list of strings. In our case, we could have just split on commas and been fine, but csv.reader also takes care of managing quotation marks and various other nuances of the comma-separated value format.
We then loop over these lines and convert them to a tuple of color and name, where the color is a tuple of three floating value integers. This tuple is constructed using a generator expression. There might be more readable ways to construct this tuple; do you think the code brevity and the speed of a generator expression is worth the obfuscation? Instead of returning a list of color tuples, it yields them one at a time, thus constructing a generator object.
Now, we need a hundred random colors. There are so many ways this can be done:
- A list comprehension with a nested generator expression:
[tuple(random() for r in range(3)) for r in range(100)] - A basic generator function
- A class that implements the
__iter__and__next__protocols - Push the data through a pipeline of coroutines
- Even just a basic
forloop
The generator version seems to be most readable, so let's add that function to our program:
from random import random
def generate_colors(count=100):
for i in range(count):
yield (random(), random(), random())
Notice how we parameterize the number of colors to generate. We can now reuse this function for other color-generating tasks in the future.
Now, before we do the classification step, we need a function to calculate the "distance" between two colors. Since it's possible to think of colors as being three dimensional (red, green, and blue could map to x, y, and z axes, for example), let's use a little basic math:
import math
def color_distance(color1, color2):
channels = zip(color1, color2)
sum_distance_squared = 0
for c1, c2 in channels:
sum_distance_squared += (c1 - c2) ** 2
return math.sqrt(sum_distance_squared)This is a pretty basic-looking function; it doesn't look like it's even using the iterator protocol. There's no yield function, no comprehensions. However, there is a for loop, and that call to the zip function is doing some real iteration as well (remember that zip yields tuples containing one element from each input iterator).
Note, however, that this function is going to be called a lot of times inside our k-nearest neighbors algorithm. If our code ran too slow and we were able to identify this function as a bottleneck, we might want to replace it with a less readable, but more optimized, generator expression:
def color_distance(color1, color2):
return math.sqrt(sum((x[0] - x[1]) ** 2 for x in zip(
color1, color2)))However, I strongly recommend not making such optimizations until you have proven that the readable version is too slow.
Now that we have some plumbing in place, let's do the actual k-nearest neighbor implementation. This seems like a good place to use a coroutine. Here it is with some test code to ensure it's yielding sensible values:
def nearest_neighbors(model_colors, num_neighbors):
model = list(model_colors)
target = yield
while True:
distances = sorted(
((color_distance(c[0], target), c) for c in model),
)
target = yield [
d[1] for d in distances[0:num_neighbors]
]
model_colors = load_colors(dataset_filename)
target_colors = generate_colors(3)
get_neighbors = nearest_neighbors(model_colors, 5)
next(get_neighbors)
for color in target_colors:
distances = get_neighbors.send(color)
print(color)
for d in distances:
print(color_distance(color, d[0]), d[1])The coroutine accepts two arguments, the list of colors to be used as a model, and the number of neighbors to query. It converts the model to a list because it's going to be iterated over multiple times. In the body of the coroutine, it accepts a tuple of RGB color values using the yield syntax. Then it combines a call to sorted with an odd generator expression. See if you can figure out what that generator expression is doing.
It returns a tuple of (distance, color_data) for each color in the model. Remember, the model itself contains tuples of (color, name), where color is a tuple of three RGB values. Therefore, the generator is returning an iterator over a weird data structure that looks like this:
(distance, (r, g, b), color_name)
The sorted call then sorts the results by their first element, which is distance. This is a complicated piece of code and isn't object-oriented at all. You may want to break it down into a normal for loop to ensure you understand what the generator expression is doing. It might also be a good exercise to imagine how this code would look if you were to pass a key argument into the sorted function instead of constructing a tuple.
The yield statement is a bit less complicated; it pulls the second value from each of the first k (distance, color_data) tuples. In more concrete terms, it yields the ((r, g, b), color_name) tuple for the k values with the lowest distance. Or, if you prefer more abstract terms, it yields the target's k-nearest neighbors in the given model.
The remaining code is just boilerplate to test this method; it constructs the model and a color generator, primes the coroutine, and prints the results in a for loop.
The two remaining tasks are to choose a color based on the nearest neighbors, and to output the results to a CSV file. Let's make two more coroutines to take care of these tasks. Let's do the output first because it can be tested independently:
def write_results(filename="output.csv"):
with open(filename, "w") as file:
writer = csv.writer(file)
while True:
color, name = yield
writer.writerow(list(color) + [name])
results = write_results()
next(results)
for i in range(3):
print(i)
results.send(((i, i, i), i * 10))This coroutine maintains an open file as state and writes lines of code to it as they are sent in using send(). The test code ensures the coroutine is working correctly, so now we can connect the two coroutines with a third one.
The second coroutine uses a bit of an odd trick:
from collections import Counter def name_colors(get_neighbors): color = yield while True: near = get_neighbors.send(color) name_guess = Counter( n[1] for n in near).most_common(1)[0][0] color = yield name_guess
This coroutine accepts, as its argument, an existing coroutine. In this case, it's an instance of nearest_neighbors. This code basically proxies all the values sent into it through that nearest_neighbors instance. Then it does some processing on the result to get the most common color out of the values that were returned. In this case, it would probably make just as much sense to adapt the original coroutine to return a name, since it isn't being used for anything else. However, there are many cases where it is useful to pass coroutines around; this is how we do it.
Now all we have to do is connect these various coroutines and pipelines together, and kick off the process with a single function call:
def process_colors(dataset_filename="colors.csv"):
model_colors = load_colors(dataset_filename)
get_neighbors = nearest_neighbors(model_colors, 5)
get_color_name = name_colors(get_neighbors)
output = write_results()
next(output)
next(get_neighbors)
next(get_color_name)
for color in generate_colors():
name = get_color_name.send(color)
output.send((color, name))
process_colors()So, this function, unlike almost every other function we've defined, is a perfectly normal function without any yield statements. It doesn't get turned into a coroutine or generator object. It does, however, construct a generator and three coroutines. Notice how the get_neighbors coroutine is passed into the constructor for name_colors? Pay attention to how all three coroutines are advanced to their first yield statements by calls to next.
Once all the pipes are created, we use a for loop to send each of the generated colors into the get_color_name coroutine, and then we pipe each of the values yielded by that coroutine to the output coroutine, which writes it to a file.
And that's it! I created a second Kivy app that loads the resulting CSV file and presents the colors to the user. The user can select either Yes or No depending on whether they think the choice made by the machine-learning algorithm matches the choice they would have made. This is not scientifically accurate (it's ripe for observation bias), but it's good enough for playing around. Using my eyes, it succeeded about 84 percent of the time, which is better than my grade 12 average. Not bad for our first ever machine learning experience, eh?
You might be wondering, "what does this have to do with object-oriented programming? There isn't even one class in this code!". In some ways, you'd be right; neither coroutines nor generators are commonly considered object-oriented. However, the functions that create them return objects; in fact, you could think of those functions as constructors. The constructed object has appropriate send() and __next__() methods. Basically, the coroutine/generator syntax is a syntax shortcut for a particular kind of object that would be quite verbose to create without it.
This case study has been an exercise in bottom-up design. We created various low-level objects that did specific tasks and hooked them all together at the end. I find this to be a common practice when developing with coroutines. The alternative, top-down design sometimes results in more monolithic pieces of code instead of unique individual pieces. In general, we want to find a happy medium between methods that are too large and methods that are too small and it's hard to see how they fit together. This is true, of course, regardless of whether the iterator protocol is being used as we did here.
If you don't use comprehensions in your daily coding very often, the first thing you should do is search through some existing code and find some for loops. See if any of them can be trivially converted to a generator expression or a list, set, or dictionary comprehension.
Test the claim that list comprehensions are faster than for loops. This can be done with the built-in timeit module. Use the help documentation for the timeit.timeit function to find out how to use it. Basically, write two functions that do the same thing, one using a list comprehension, and one using a for loop. Pass each function into timeit.timeit, and compare the results. If you're feeling adventurous, compare generators and generator expressions as well. Testing code using timeit can become addictive, so bear in mind that code does not need to be hyperfast unless it's being executed an immense number of times, such as on a huge input list or file.
Play around with generator functions. Start with basic iterators that require multiple values (mathematical sequences are canonical examples; the Fibonacci sequence is overused if you can't think of anything better). Try some more advanced generators that do things like take multiple input lists and somehow yield values that merge them. Generators can also be used on files; can you write a simple generator that shows those lines that are identical in two files?
Coroutines abuse the iterator protocol but don't actually fulfill the iterator pattern. Can you build a non-coroutine version of the code that gets a serial number from a log file? Take an object-oriented approach so that you can store an additional state on a class. You'll learn a lot about coroutines if you can create an object that is a drop-in replacement for the existing coroutine.
See if you can abstract the coroutines used in the case study so that the k-nearest-neighbor algorithm can be used on a variety of datasets. You'll likely want to construct a coroutine that accepts other coroutines or functions that do the distance and recombination calculations as parameters, and then calls into those functions to find the actual nearest neighbors.
In this chapter, we learned that design patterns are useful abstractions that provide "best practice" solutions for common programming problems. We covered our first design pattern, the iterator, as well as numerous ways that Python uses and abuses this pattern for its own nefarious purposes. The original iterator pattern is extremely object-oriented, but it is also rather ugly and verbose to code around. However, Python's built-in syntax abstracts the ugliness away, leaving us with a clean interface to these object-oriented constructs.
Comprehensions and generator expressions can combine container construction with iteration in a single line. Generator objects can be constructed using the yield syntax. Coroutines look like generators on the outside but serve a much different purpose.
We'll cover several more design patterns in the next two chapters.