Introducing regular expressions

A regular expression, or regex, is a pattern to match text. In other words, it allows us to define an abstract string (typically, the definition of a structured kind of text) to check against other strings to see if they match or not.

It is better to describe them with an example. Think of defining a pattern of text, such as a word that starts with an uppercase A and contains only lowercase 'n's and 'a's after that. Let's show some possible comparisons and results:

Table 1.1: A pattern matching example

If this sounds complicated, that's because it is. Regexes can, notoriously, be complicated, intricate and difficult to follow. But they are also very useful because they allow us to perform incredibly powerful pattern matching.

Some common uses of regexes are:

• Validating input data: For example, a phone number consisting only of numbers, dashes, and brackets.
• String parsing: Retrieve data from structured strings, such as logs or URLs. Typically, capture some section of the full string. This is similar to what's described in the previous recipe.
• Scraping: Find the occurrences of something in a long piece of text. For example, find all the email addresses in a web page.
• Replacement: Find and replace a word or words with others. For example, replace the owner with John Smith.

Getting ready

The Python module to deal with regexes is called re. The main function we'll cover is re.search(), which returns a match object with information about what matched the pattern.

Some characters are special and refer to concepts such as the end of the string, any digit, any character, any whitespace character, and so on.

The simplest form is just a literal string. For example, the regex pattern r'LOG' matches the string 'LOGS', but not the string 'NOT A MATCH'. If there's no match, re.search returns None. If there is, it returns a special Match object:

>>> import re
>>> re.search(r'LOG', 'LOGS')
<_sre.SRE_Match object; span=(0, 3), match='LOG'>
>>> re.search(r'LOG', 'NOT A MATCH')
>>>

How to do it…

1. Import the re module:

   >>> import re

2. Then, to match a pattern that is not at the start of the string:

   >>> re.search(r'LOG', 'SOME LOGS')
   <_sre.SRE_Match object; span=(5, 8), match='LOG'>

3. Match a pattern that is only at the start of the string. Note the ^ character at the start of the pattern:

   >>> re.search(r'^LOG', 'LOGS')
   <_sre.SRE_Match object; span=(0, 3), match='LOG'>
   >>> re.search(r'^LOG', 'SOME LOGS')
   >>>

4. Match a pattern only at the end of the string. Note the $ character at the end:

   >>> re.search(r'LOG$', 'SOME LOG')
   <_sre.SRE_Match object; span=(5, 8), match='LOG'>
   >>> re.search(r'LOG$', 'SOME LOGS')
   >>>

5. Match the word 'thing' (not excluding things), but not something or anything. Note the \b at the start of the second pattern:

   >>> STRING = 'something in the things she shows me'
   >>> match = re.search(r'thing', STRING)
   >>> STRING[:match.start()], STRING[match.start():match.end()], STRING[match.end():]
   ('some', 'thing', ' in the things she shows me')
   >>> match = re.search(r'\bthing', STRING)
   >>> STRING[:match.start()], STRING[match.start():match.end()], STRING[match.end():]
   ('something in the ', 'thing', 's she shows me')

6. Match a pattern that's only numbers and dashes (for example, a phone number). Retrieve the matched string:

   >>> re.search(r'[0123456789-]+', 'the phone number is 1234-567-890') <_sre.SRE_Match object; span=(20, 32), match='1234-567-890'>
   >>> re.search(r'[0123456789-]+', 'the phone number is 1234-567-890').group()
   '1234-567-890'

7. Match an email address naively:

   >>> re.search(r'\S+@\S+', 'my email is [email protected]').group() '[email protected]'

How it works…

The re.search function matches a pattern, no matter its position in the string. As explained previously, this will return None if the pattern is not found, or a Match object.

The following special characters are used:

• ^: Marks the start of the string
• $: Marks the end of the string
• \b: Marks the start or end of a word
• \S: Marks any character that's not a whitespace, including characters like * or $

More special characters are shown in the next recipe, Going deeper into regular expressions.

In step 6 of the How to do it… section, the r'[0123456789‑]+' pattern is composed of two parts. The first is between square brackets, and matches any single character between 0 and 9 (any number) and the dash (‑) character. The + sign after that means that this character can be present one or more times. This is called a quantifier in regexes. This makes a match on any combination of numbers and dashes, no matter how long it is.

Step 7 again uses the + sign to match as many characters as necessary before the @ and again after it. In this case, the character match is \S, which matches any non-whitespace character.

The resulting matching object returns the position where the matched pattern starts and ends (using the start and end methods), as shown in step 5, which splits the string into matched parts, showing the distinction between the two matching patterns.

The specific matched pattern can be retrieved by calling group(), as shown in step 6. Note that the result will always be a string. It can be further processed using any of the methods we've seen previously, such as by splitting the phone number into groups by dashes, for example:

>>> match = re.search(r'[0123456789-]+', 'the phone number is 1234-567-890')
>>> [int(n) for n in match.group().split('-')]
[1234, 567, 890]

The captured string will include all characters, like the dashes in this case. Note how we remove them by using .split()

There's more…

Dealing with regexes can be difficult and complex. Please allow time to test your matches and be sure that they work as you expect in order to avoid nasty surprises.

"Some people, when confronted with a problem, think "I know, I'll use regular expressions." Now they have two problems."

– Jamie Zawinski

Regular expressions are at their best when they are kept very simple. In general, if there is a specific tool to do it, prefer it over regexes. A very clear example of this is with HTML parsing; refer to Chapter 3, Building Your First Web Scraping Application, for better tools to achieve this.

You can check your regexes interactively. A good one that's freely available online is https://regex101.com/, which displays each of the elements and explains the regex. Double-check that you're using the Python flavor:

Figure 1.2: An example using RegEx101

Note that the EXPLANATION box in the preceding image describes that \b matches a word boundary (the start or end of a word), and that thing matches literally these characters.

Regexes, in some cases, can be very slow, or even susceptible to what's called a regex denial-of-service attack, a string created to confuse a particular regex so that it takes an enormous amount of time. In the worst-case scenario, it can even block the computer. While automating tasks probably won't get you into those problems, keep an eye out in case a regex takes too long to process.

See also

• The Extracting data from structured strings recipe, earlier in the chapter, to learn simple techniques to extract information from text.
• The Using a third-party tool—parse recipe, earlier in the chapter, to use a third-party tool to extract information from text.
• The Going deeper into regular expressions recipe, covered next, to further your knowledge of regular expressions.