Chapter 1: Benchmarking and Profiling

Recognizing the slow parts of your program is the single most important task when it comes to speeding up your code. In most cases, the code that causes the application to slow down is a very small fraction of the program. By identifying these critical sections, you can focus on the parts that need the most improvement without wasting time in micro-optimization.

Profiling is a technique that allows us to pinpoint the most resource-intensive parts of an application. A profiler is a program that runs an application and monitors how long each function takes to execute, thus detecting the functions on which your application spends most of its time.

Python provides several tools to help us find these bottlenecks and measure important performance metrics. In this chapter, we will learn how to use the standard cProfile module and the line_profiler third-party package. We will also learn how to profile the memory consumption of an application through the memory_profiler tool. Another useful tool that we will cover is KCachegrind, which can be used to graphically display the data produced by various profilers.

Finally, benchmarks are small scripts used to assess the total execution time of your application. We will learn how to write benchmarks and use them to accurately time your programs.

The topics we will cover in this chapter are listed here:

• Designing your application
• Writing tests and benchmarks
• Writing better tests and benchmarks with pytest-benchmark
• Finding bottlenecks with cProfile
• Optimizing our code
• Using the dis module
• Profiling memory usage with memory_profiler

By the end of the chapter, you will have gained a solid understanding of how to optimize a Python program and will be armed with practical tools that facilitate the optimization process.