Book Image

Modern Python Cookbook - Second Edition

By : Steven F. Lott
Book Image

Modern Python Cookbook - Second Edition

By: Steven F. Lott

Overview of this book

Python is the preferred choice of developers, engineers, data scientists, and hobbyists everywhere. It is a great language that can power your applications and provide great speed, safety, and scalability. It can be used for simple scripting or sophisticated web applications. By exposing Python as a series of simple recipes, this book gives you insight into specific language features in a particular context. Having a tangible context helps make the language or a given standard library feature easier to understand. This book comes with 133 recipes on the latest version of Python 3.8. The recipes will benefit everyone, from beginners just starting out with Python to experts. You'll not only learn Python programming concepts but also how to build complex applications. The recipes will touch upon all necessary Python concepts related to data structures, object oriented programming, functional programming, and statistical programming. You will get acquainted with the nuances of Python syntax and how to effectively take advantage of it. By the end of this Python book, you will be equipped with knowledge of testing, web services, configuration, and application integration tips and tricks. You will be armed with the knowledge of how to create applications with flexible logging, powerful configuration, command-line options, automated unit tests, and good documentation.
Table of Contents (18 chapters)
Other Books You May Enjoy

String parsing with regular expressions

How do we decompose a complex string? What if we have complex, tricky punctuation? Or—worse yet—what if we don't have punctuation, but have to rely on patterns of digits to locate meaningful information?

Getting ready

The easiest way to decompose a complex string is by generalizing the string into a pattern and then writing a regular expression that describes that pattern.

There are limits to the patterns that regular expressions can describe. When we're confronted with deeply nested documents in a language like HTML, XML, or JSON, we often run into problems, and can't use regular expressions.

The re module contains all of the various classes and functions we need to create and use regular expressions.

Let's say that we want to decompose text from a recipe website. Each line looks like this:

>>> ingredient = "Kumquat: 2 cups"

We want to separate the ingredient from the measurements.

How to do it...

To write and use regular expressions, we often do this:

  1. Generalize the example. In our case, we have something that we can generalize as:
    (ingredient words): (amount digits) (unit words)
  2. We've replaced literal text with a two-part summary: what it means and how it's represented. For example, ingredient is represented as words, while amount is represented as digits. Import the re module:
    >>> import re
  3. Rewrite the pattern into Regular expression (RE) notation:
    >>> ingredient_pattern = re.compile(r'([\w\s]+):\s+(\d+)\s+(\w+)')

    We've replaced representation hints such as ingredient words, a mixture of letters and spaces, with [\w\s]+. We've replaced amount digits with \d+. And we've replaced single spaces with \s+ to allow one or more spaces to be used as punctuation. We've left the colon in place because, in the regular expression notation, a colon matches itself.

    For each of the fields of data, we've used () to capture the data matching the pattern. We didn't capture the colon or the spaces because we don't need the punctuation characters.

    REs typically use a lot of \ characters. To make this work out nicely in Python, we almost always use raw strings. The r' prefix tells Python not to look at the \ characters and not to replace them with special characters that aren't on our keyboards.

  4. Compile the pattern:
    >>> pattern = re.compile(pattern_text)
  5. Match the pattern against the input text. If the input matches the pattern, we'll get a match object that shows details of the matching:
    >>> match = pattern.match(ingredient)
    >>> match is None
    >>> match.groups()
     ('Kumquat', '2', 'cups')
  6. Extract the named groups of characters from the match object:

Each group is identified by the order of the capture ()s in the regular expression. This gives us a tuple of the different fields captured from the string. We'll return to the use of tuples in the Using tuples recipe. This can be confusing in more complex regular expressions; there is a way to provide a name, instead of the numeric position, to identify a capture group.

How it works...

There are a lot of different kinds of string patterns that we can describe with RE.

We've shown a number of character classes:

  • \w matches any alphanumeric character (a to z, A to Z, 0 to 9)
  • \d matches any decimal digit
  • \s matches any space or tab character

These classes also have inverses:

  • \W matches any character that's not a letter or a digit
  • \D matches any character that's not a digit
  • \S matches any character that's not some kind of space or tab

Many characters match themselves. Some characters, however, have a special meaning, and we have to use \ to escape from that special meaning:

  • We saw that + as a suffix means to match one or more of the preceding patterns. \d+ matches one or more digits. To match an ordinary +, we need to use \+.
  • We also have * as a suffix, which matches zero or more of the preceding patterns. \w* matches zero or more characters. To match a *, we need to use \*.
  • We have ? as a suffix, which matches zero or one of the preceding expressions. This character is used in other places, and has a different meaning in the other context. We'll see it used in (?P<name>...), where it is inside () to define special properties for the grouping.
  • . matches any single character. To match a . specifically, we need to use \..

We can create our own unique sets of characters using [] to enclose the elements of the set. We might have something like this:


This has a \w+ to match any number of alphanumeric characters. This will be collected into a group called name.

It uses \s* to match an optional sequence of spaces.

It matches any character in the set [=:]. Exactly one of the characters in this set must be present.

It uses \s* again to match an optional sequence of spaces.

Finally, it uses .* to match everything else in the string. This is collected into a group named value.

We can use this to parse strings, like this:

    size = 12 
    weight: 14

By being flexible with the punctuation, we can make a program easier to use. We'll tolerate any number of spaces, and either an = or a : as a separator.

There's more...

A long regular expression can be awkward to read. We have a clever Pythonic trick for presenting an expression in a way that's much easier to read:

>>> ingredient_pattern = re.compile(
... r'(?P<ingredient>[\w\s]+):\s+' # name of the ingredient up to the ":"
... r'(?P<amount>\d+)\s+'          # amount, all digits up to a space
... r'(?P<unit>\w+)'               # units, alphanumeric characters
... )

This leverages three syntax rules:

  • A statement isn't finished until the () characters match.
  • Adjacent string literals are silently concatenated into a single long string.
  • Anything between # and the end of the line is a comment, and is ignored.

We've put Python comments after the important clauses in our regular expression. This can help us understand what we did, and perhaps help us diagnose problems later.

We can also use the regular expression's "verbose" mode to add gratuitous whitespace and comments inside a regular expression string. To do this, we must use re.X as an option when compiling a regular expression to make whitespace and comments possible. This revised syntax looks like this:

>>> ingredient_pattern_x = re.compile(r'''
... (?P<ingredient>[\w\s]+):\s+ # name of the ingredient up to the ":"'
... (?P<amount>\d+)\s+          # amount, all digits up to a space'
... (?P<unit>\w+)               # units, alphanumeric characters
... ''', re.X)

We can either break the pattern up or make use of extended syntax to make the regular expression more readable.

See also