The easiest way to decompose a complex string is by generalizing the string into a pattern and then writing a regular expression that describes that pattern.

There are limits to the patterns that regular expressions can describe. When we're confronted with deeply-nested documents in a language like HTML, XML, or JSON, we often run into problems, and can't use regular expressions.

The re module contains all of the various classes and functions we need to create and use regular expressions.

Let's say that we want to decompose text from a recipe website. Each line looks like this:

>>> ingredient = "Kumquat: 2 cups"

We want to separate the ingredient from the measurements.

To write and use regular expressions, we often do this:

(ingredient words): (amount digits) (unit words)

>>> import re

>>> pattern_text = r'(?P<ingredient>\w+):\s+(?P<amount>\d+)\s+(?P<unit>\w+)'

We've replaced representation hints such as words with \w+. We've replaced digits with \d+. And we've replaced single spaces with \s+ to allow one or more spaces to be used as punctuation. We've left the colon in place, because in the regular expression notation, a colon matches itself.

For each of the fields of data, we've used ?P<name> to provide a name that identifies the data we want to extract. We didn't do this around the colon or the spaces because we don't want those characters.

REs use a lot of \ characters. To make this work out nicely in Python, we almost always use raw strings. The r' prefix tells Python not to look at the \ characters and not to replace them with special characters that aren't on our keyboards.

>>> pattern = re.compile(pattern_text)

>>> match = pattern.match(ingredient)>>> match is NoneFalse>>> match.groups()('Kumquat', '2', 'cups')

This, by itself, is pretty cool: we have a tuple of the different fields within the string. We'll return to the use of tuples in a recipe named Using tuples.

>>> match.group('ingredient')'Kumquat'>>> match.group('amount')'2'>>> match.group('unit')'cups'

Each group is identified by the name we used in the (?P<name>...) part of the RE.

There are a lot of different kinds of string patterns that we can describe with RE.

We've shown a number of character classes:

These classes also have inverses:

Many characters match themselves. Some characters, however, have special meaning, and we have to use \ to escape from that special meaning:

We can create our own unique sets of characters using [] to enclose the elements of the set. We might have something like this:

    (?P<name>\w+)\s*[=:]\s*(?P<value>.*)

This has a \w+ to match any number of alphanumeric characters. This will be collected into a group with the name of name.

It uses \s* to match an optional sequence of spaces.

It matches any character in the set [=:]. One of the two characters in this set must be present.

It uses \s* again to match an optional sequence of spaces.

Finally, it uses .* to match everything else in the string. This is collected into a group named value.

We can use this to parse strings like this:

    size = 12 
    weight: 14

By being flexible with the punctuation, we can make a program easier to use. We'll tolerate any number of spaces, and either an = or a : as a separator.

A long regular expression can be awkward to read. We have a clever Pythonic trick for presenting an expression in a way that's much easier to read:

>>> ingredient_pattern = re.compile(... r'(?P<ingredient>\w+):\s+' # name of the ingredient up to the ":"... r'(?P<amount>\d+)\s+'      # amount, all digits up to a space... r'(?P<unit>\w+)'           # units, alphanumeric characters... )

This leverages three syntax rules:

We've put Python comments after the important clauses in our regular expression. This can help us understand what we did, and perhaps help us diagnose problems later.