Book Image

Secret Recipes of the Python Ninja

Book Image

Secret Recipes of the Python Ninja

Overview of this book

This book covers the unexplored secrets of Python, delve into its depths, and uncover its mysteries. You’ll unearth secrets related to the implementation of the standard library, by looking at how modules actually work. You’ll understand the implementation of collections, decimals, and fraction modules. If you haven’t used decorators, coroutines, and generator functions much before, as you make your way through the recipes, you’ll learn what you’ve been missing out on. We’ll cover internal special methods in detail, so you understand what they are and how they can be used to improve the engineering decisions you make. Next, you’ll explore the CPython interpreter, which is a treasure trove of secret hacks that not many programmers are aware of. We’ll take you through the depths of the PyPy project, where you’ll come across several exciting ways that you can improve speed and concurrency. Finally, we’ll take time to explore the PEPs of the latest versions to discover some interesting hacks.
Table of Contents (17 chapters)
Title Page
Copyright and Credits
Packt Upsell
Foreword
Contributors
Preface
Index

How to implement multiprocessing


Multiprocessing in Python involves starting separate processes, much like forking. This gets around the GIL and its effect on multiple threads, but you have to deal with the overhead of increased memory usage and the multiple instances of the Python interpreter that are spawned for all the processes. However, in multi-core systems, multiprocessing can take advantage of the different CPUs so you have true parallelism; more cores = more processing power.

As there isn't room to cover everything about parallel Python programming (there are entire books written on the subject), I'm going to finish this chapter by demonstrating how to automate multiprocessing using Pool(), which controls worker processes automatically. Pool() accepts a number of input arguments, probably the most important one being the number of processes. By default, Pool() uses all the available CPUs on your system. This is useful because, if your system is upgraded, your program will automatically...