Chapter 1: Benchmarking and Profiling
Recognizing the slow parts of your program is the single most important task when it comes to speeding up your code. In most cases, the code that causes the application to slow down is a very small fraction of the program. By identifying these critical sections, you can focus on the parts that need the most improvement without wasting time in micro-optimization.
Profiling is a technique that allows us to pinpoint the most resource-intensive parts of an application. A profiler is a program that runs an application and monitors how long each function takes to execute, thus detecting the functions on which your application spends most of its time.
Python provides several tools to help us find these bottlenecks and measure important performance metrics. In this chapter, we will learn how to use the standard cProfile module and the line_profiler third-party package. We will also learn how to profile the memory consumption of an application through the memory_profiler tool. Another useful tool that we will cover is KCachegrind, which can be used to graphically display the data produced by various profilers.
Finally, benchmarks are small scripts used to assess the total execution time of your application. We will learn how to write benchmarks and use them to accurately time your programs.
The topics we will cover in this chapter are listed here:
- Designing your application
- Writing tests and benchmarks
- Writing better tests and benchmarks with
pytest-benchmark - Finding bottlenecks with
cProfile - Optimizing our code
- Using the
dismodule - Profiling memory usage with
memory_profiler
By the end of the chapter, you will have gained a solid understanding of how to optimize a Python program and will be armed with practical tools that facilitate the optimization process.