Book Image

Modern Python Cookbook - Second Edition

By : Steven F. Lott
Book Image

Modern Python Cookbook - Second Edition

By: Steven F. Lott

Overview of this book

Python is the preferred choice of developers, engineers, data scientists, and hobbyists everywhere. It is a great language that can power your applications and provide great speed, safety, and scalability. It can be used for simple scripting or sophisticated web applications. By exposing Python as a series of simple recipes, this book gives you insight into specific language features in a particular context. Having a tangible context helps make the language or a given standard library feature easier to understand. This book comes with 133 recipes on the latest version of Python 3.8. The recipes will benefit everyone, from beginners just starting out with Python to experts. You'll not only learn Python programming concepts but also how to build complex applications. The recipes will touch upon all necessary Python concepts related to data structures, object oriented programming, functional programming, and statistical programming. You will get acquainted with the nuances of Python syntax and how to effectively take advantage of it. By the end of this Python book, you will be equipped with knowledge of testing, web services, configuration, and application integration tips and tricks. You will be armed with the knowledge of how to create applications with flexible logging, powerful configuration, command-line options, automated unit tests, and good documentation.
Table of Contents (18 chapters)
Other Books You May Enjoy

Encoding strings – creating ASCII and UTF-8 bytes

Our computer files are bytes. When we upload or download from the internet, the communication works in bytes. A byte only has 256 distinct values. Our Python characters are Unicode. There are a lot more than 256 Unicode characters.

How do we map Unicode characters to bytes to write to a file or for transmission?

Getting ready

Historically, a character occupied 1 byte. Python leverages the old ASCII encoding scheme for bytes; this sometimes leads to confusion between bytes and proper strings of Unicode characters.

Unicode characters are encoded into sequences of bytes. There are a number of standardized encodings and a number of non-standard encodings.

Plus, there also are some encodings that only work for a small subset of Unicode characters. We try to avoid these, but there are some situations where we'll need to use a subset encoding scheme.

Unless we have a really good reason not to, we almost always use UTF-8 encoding for Unicode characters. Its main advantage is that it's a compact representation of the Latin alphabet, which is used for English and a number of European languages.

Sometimes, an internet protocol requires ASCII characters. This is a special case that requires some care because the ASCII encoding can only handle a small subset of Unicode characters.

How to do it...

Python will generally use our OS's default encoding for files and internet traffic. The details are unique to each OS:

  1. We can make a general setting using the PYTHONIOENCODING environment variable. We set this outside of Python to ensure that a particular encoding is used everywhere. When using Linux or macOS, use export to set the environment variable. For Windows, use the set command, or the PowerShell Set-Item cmdlet. For Linux, it looks like this:
  2. Run Python:
  3. We sometimes need to make specific settings when we open a file inside our script. We'll return to this topic in Chapter 10, Input/Output, Physical Format and, Logical Layout. Open the file with a given encoding. Read or write Unicode characters to the file:
    >>> with open('some_file.txt', 'w', encoding='utf-8') as output:
    ...     print( 'You drew \U0001F000', file=output )
    >>> with open('some_file.txt', 'r', encoding='utf-8') as input:
    ...     text =
    >>> text
    'You drew '

We can also manually encode characters, in the rare case that we need to open a file in bytes mode; if we use a mode of wb, we'll need to use manual encoding:

>>> string_bytes = 'You drew \U0001F000'.encode('utf-8')
>>> string_bytes
b'You drew \xf0\x9f\x80\x80'

We can see that a sequence of bytes (\xf0\x9f\x80\x80) was used to encode a single Unicode character, U+1F000, .

How it works...

Unicode defines a number of encoding schemes. While UTF-8 is the most popular, there is also UTF-16 and UTF-32. The number is the typical number of bits per character. A file with 1,000 characters encoded in UTF-32 would be 4,000 8-bit bytes. A file with 1,000 characters encoded in UTF-8 could be as few as 1,000 bytes, depending on the exact mix of characters. In UTF-8 encoding, characters with Unicode numbers above U+007F require multiple bytes.

Various OSes have their own coding schemes. macOS files can be encoded in Mac Roman or Latin-1. Windows files might use CP1252 encoding.

The point with all of these schemes is to have a sequence of bytes that can be mapped to a Unicode character and—going the other way—a way to map each Unicode character to one or more bytes. Ideally, all of the Unicode characters are accounted for. Pragmatically, some of these coding schemes are incomplete.

The historical form of ASCII encoding can only represent about 100 of the Unicode characters as bytes. It's easy to create a string that cannot be encoded using the ASCII scheme.

Here's what the error looks like:

>>> 'You drew \U0001F000'.encode('ascii')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character '\U0001f000' in position 9: ordinal not in range(128)

We may see this kind of error when we accidentally open a file with a poorly chosen encoding. When we see this, we'll need to change our processing to select a more useful encoding; ideally, UTF-8.

Bytes versus strings: Bytes are often displayed using printable characters. We'll see b'hello' as shorthand for a five-byte value. The letters are chosen using the old ASCII encoding scheme, where byte values from 0x20 to 0x7F will be shown as characters, and outside this range, more complex-looking escapes will be used.

This use of characters to represent byte values can be confusing. The prefix of b' is our hint that we're looking at bytes, not proper Unicode characters.

See also

  • There are a number of ways to build strings of data. See the Building complex strings with f"strings" and the Building complex strings from lists of characters recipes for examples of creating complex strings. The idea is that we might have an application that builds a complex string, and then we encode it into bytes.
  • For more information on UTF-8 encoding, see
  • For general information on Unicode encodings, see