Book Image

Mastering Concurrency in Python

By : Quan Nguyen
Book Image

Mastering Concurrency in Python

By: Quan Nguyen

Overview of this book

Python is one of the most popular programming languages, with numerous libraries and frameworks that facilitate high-performance computing. Concurrency and parallelism in Python are essential when it comes to multiprocessing and multithreading; they behave differently, but their common aim is to reduce the execution time. This book serves as a comprehensive introduction to various advanced concepts in concurrent engineering and programming. Mastering Concurrency in Python starts by introducing the concepts and principles in concurrency, right from Amdahl's Law to multithreading programming, followed by elucidating multiprocessing programming, web scraping, and asynchronous I/O, together with common problems that engineers and programmers face in concurrent programming. Next, the book covers a number of advanced concepts in Python concurrency and how they interact with the Python ecosystem, including the Global Interpreter Lock (GIL). Finally, you'll learn how to solve real-world concurrency problems through examples. By the end of the book, you will have gained extensive theoretical knowledge of concurrency and the ways in which concurrency is supported by the Python language

Related Content you might be interested in

Current Title:

Small book image
Mastering Concurrency in Python
Quan Nguyen
Nov, 2018 | 446 pages
Small book image
Learning Concurrency in Python
Elliot Forbes
Aug, 2017 | 360 pages
Small book image
Python Parallel Programming Cookbook
Giancarlo Zaccone
Sep, 2019 | 370 pages
Small book image
Crafting Test-Driven Software with Python
Alessandro Molina
Feb, 2021 | 338 pages
Table of Contents (22 chapters)
Preface
Preface
Who this book is for
What this book covers
To get the most out of this book
Get in touch
Free Chapter
1
Advanced Introduction to Concurrent and Parallel Programming
Advanced Introduction to Concurrent and Parallel Programming
Technical requirements
What is concurrency?
Not everything should be made concurrent
The history, present, and future of concurrency
A brief overview of mastering concurrency in Python
Setting up your Python environment
Summary
Questions
Further reading
2
Amdahl's Law
Amdahl's Law
Technical requirements
Amdahl's Law
Formula and interpretation
Amdahl's Law's relationship to the law of diminishing returns
How to simulate in Python
Practical applications of Amdahl's Law
Summary
Questions
Further reading
3
Working with Threads in Python
Working with Threads in Python
Technical requirements
The concept of a thread
An overview of the threading module
Creating a new thread in Python
Synchronizing threads
Multithreaded priority queue
Summary
Questions
Further reading
4
Using the with Statement in Threads
Using the with Statement in Threads
Technical requirements
Context management
The with statement in concurrent programming
Summary
Questions
Further reading
5
Concurrent Web Requests
Concurrent Web Requests
Technical requirements
The basics of web requests
The requests module
Concurrent web requests
The problem of timeout
Good practices in making web requests
Summary
Questions
Further reading
6
Working with Processes in Python
Working with Processes in Python
Technical requirements
The concept of a process
An overview of the multiprocessing module
Interprocess communication
Summary
Questions
Further reading
7
Reduction Operators in Processes
Reduction Operators in Processes
Technical requirements
The concept of reduction operators
Example implementation in Python
Real-life applications of concurrent reduction operators
Summary
Questions
Further reading
8
Concurrent Image Processing
Concurrent Image Processing
Technical requirements
Image processing fundamentals
Applying concurrency to image processing
Good concurrent image processing practices
Summary
Questions
Further reading
9
Introduction to Asynchronous Programming
Introduction to Asynchronous Programming
Technical requirements
A quick analogy
Asynchronous versus other programming models
An example in Python
Summary
Questions
Further reading
10
Implementing Asynchronous Programming in Python
Implementing Asynchronous Programming in Python
Technical requirements
The asyncio module
The asyncio framework in action
concurrent.futures as a solution for blocking tasks
Summary
Questions
Further reading
11
Building Communication Channels with asyncio
Building Communication Channels with asyncio
Technical requirements
The ecosystem of communication channels
Python example
Client-side communication with aiohttp
Summary
Questions
Further reading
12
Deadlocks
Deadlocks
Technical requirements
The concept of deadlock
Approaches to deadlock situations
The concept of livelock
Summary
Questions
Further reading
13
Starvation
Starvation
Technical requirements
The concept of starvation
The readers-writers problem
Solutions to starvation
Summary
Questions
Further reading
14
Race Conditions
Race Conditions
Technical requirements
The concept of race conditions
Simulating race conditions in Python
Locks as a solution to race conditions
Race conditions in real life
Summary
Questions
Further reading
15
The Global Interpreter Lock
The Global Interpreter Lock
Technical requirements
An introduction to the Global Interpreter Lock
The potential removal of the GIL from Python
How to work with the GIL
Summary
Questions
Further reading
16
Designing Lock-Based and Mutex-Free Concurrent Data Structures
Designing Lock-Based and Mutex-Free Concurrent Data Structures
Technical requirements
Lock-based concurrent data structures in Python
Mutex-free concurrent data structures in Python
Building on simple data structures
Summary
Questions
Further reading
17
Memory Models and Operations on Atomic Types
Memory Models and Operations on Atomic Types
Technical requirements
Python memory model
Atomic operations in Python
Summary
Questions
Further reading
18
Building a Server from Scratch
Building a Server from Scratch
Technical requirements
Low-level network programming via the socket module
Building a calculator server with the socket module
Building a non-blocking server
Summary
Questions
Further reading
19
Testing, Debugging, and Scheduling Concurrent Applications
Testing, Debugging, and Scheduling Concurrent Applications
Technical requirements
Scheduling with APScheduler
Testing and concurrency in Python
Debugging concurrent programs
Summary
Questions
Further reading
20
Assessments
Assessments
Chapter 1
Chapter 2
Chapter 3
Chapter 4
Chapter 5
Chapter 6
Chapter 7
Chapter 8
Chapter 9
Chapter 10
Chapter 11
Chapter 12
Chapter 13
Chapter 14
Chapter 15
Chapter 16
Chapter 17
Chapter 18
Chapter 19
21
Other Books You May Enjoy
Other Books You May Enjoy
Leave a review - let other readers know what you think

How to work with the GIL

There are a few ways to deal with the GIL in your Python applications, which will be addressed as follows.

Implementing multiprocessing, rather than multithreading

This is perhaps the most popular and easiest method to circumvent the GIL and achieve optimal speed in a concurrent program. As the GIL only prevents multiple threads from executing CPU-bound tasks simultaneously, processes executing over multiple cores of a system, each having its own memory space, are completely immune to the GIL.

Specifically, considering the preceding countdown example, let's compare the performance of that CPU-bound program when it is sequential, multithreading, and multiprocessing. Navigate to the Chapter15/example3...

Previous Section
End of Section 5
Next Section