Book Image

Advanced Python Programming

By : Dr. Gabriele Lanaro, Quan Nguyen, Sakis Kasampalis
Book Image

Advanced Python Programming

By: Dr. Gabriele Lanaro, Quan Nguyen, Sakis Kasampalis

Overview of this book

This Learning Path shows you how to leverage the power of both native and third-party Python libraries for building robust and responsive applications. You will learn about profilers and reactive programming, concurrency and parallelism, as well as tools for making your apps quick and efficient. You will discover how to write code for parallel architectures using TensorFlow and Theano, and use a cluster of computers for large-scale computations using technologies such as Dask and PySpark. With the knowledge of how Python design patterns work, you will be able to clone objects, secure interfaces, dynamically choose algorithms, and accomplish much more in high performance computing. By the end of this Learning Path, you will have the skills and confidence to build engaging models that quickly offer efficient solutions to your problems. This Learning Path includes content from the following Packt products: • Python High Performance - Second Edition by Gabriele Lanaro • Mastering Concurrency in Python by Quan Nguyen • Mastering Python Design Patterns by Sakis Kasampalis

Related Content you might be interested in

Current Title:

Small book image
Advanced Python Programming
Dr. Gabriele Lanaro | Quan Nguyen | Sakis Kasampalis
Feb, 2019 | 11 hours 12 minutes
No titles found
Table of Contents (41 chapters)
Title Page
Title Page
Copyright
Copyright
About Packt
About Packt
Contributors
Contributors
Preface
Preface
Free Chapter
1
Benchmarking and Profiling
Benchmarking and Profiling
Designing your application
Writing tests and benchmarks
Better tests and benchmarks with pytest-benchmark
Finding bottlenecks with cProfile
Profile line by line with line_profiler
Optimizing our code
The dis module
Profiling memory usage with memory_profiler
Summary
2
Pure Python Optimizations
Pure Python Optimizations
Useful algorithms and data structures
Caching and memoization
Comprehensions and generators
Summary
3
Fast Array Operations with NumPy and Pandas
Fast Array Operations with NumPy and Pandas
Getting started with NumPy
Rewriting the particle simulator in NumPy
Reaching optimal performance with numexpr
Pandas
Summary
4
C Performance with Cython
C Performance with Cython
Compiling Cython extensions
Adding static types
Sharing declarations
Working with arrays
Particle simulator in Cython
Profiling Cython
Using Cython with Jupyter
Summary
5
Exploring Compilers
Exploring Compilers
Numba
The PyPy project
Other interesting projects
Summary
6
Implementing Concurrency
Implementing Concurrency
Asynchronous programming
The asyncio framework
Reactive programming
Summary
7
Parallel Processing
Parallel Processing
Introduction to parallel programming
Using multiple processes
Parallel Cython with OpenMP
Automatic parallelism
Summary
8
Advanced Introduction to Concurrent and Parallel Programming
Advanced Introduction to Concurrent and Parallel Programming
Technical requirements
What is concurrency?
Not everything should be made concurrent
The history, present, and future of concurrency
A brief overview of mastering concurrency in Python
Setting up your Python environment
Summary
Questions
Further reading
9
Amdahl's Law
Amdahl's Law
Technical requirements
Amdahl's Law
Formula and interpretation
Amdahl's Law's relationship to the law of diminishing returns
How to simulate in Python
Practical applications of Amdahl's Law
Summary
Questions
Further reading
10
Working with Threads in Python
Working with Threads in Python
Technical requirements
The concept of a thread
An overview of the threading module
Creating a new thread in Python
Synchronizing threads
Multithreaded priority queue
Summary
Questions
Further reading
11
Using the with Statement in Threads
Using the with Statement in Threads
Technical requirements
Context management
The with statement in concurrent programming
Summary
Questions
Further reading
12
Concurrent Web Requests
Concurrent Web Requests
Technical requirements
The basics of web requests
The requests module
Concurrent web requests
The problem of timeout
Good practices in making web requests
Summary
Questions
Further reading
13
Working with Processes in Python
Working with Processes in Python
Technical requirements
The concept of a process
An overview of the multiprocessing module
Interprocess communication
Summary
Questions
Further reading
14
Reduction Operators in Processes
Reduction Operators in Processes
Technical requirements
The concept of reduction operators
Example implementation in Python
Real-life applications of concurrent reduction operators
Summary
Questions
Further reading
15
Concurrent Image Processing
Concurrent Image Processing
Technical requirements
Image processing fundamentals
Applying concurrency to image processing
Good concurrent image processing practices
Summary
Questions
Further reading
16
Introduction to Asynchronous Programming
Introduction to Asynchronous Programming
Technical requirements
A quick analogy
Asynchronous versus other programming models
An example in Python
Summary
Questions
Further reading
17
Implementing Asynchronous Programming in Python
Implementing Asynchronous Programming in Python
Technical requirements
The asyncio module
The asyncio framework in action
concurrent.futures as a solution for blocking tasks
Summary
Questions
Further reading
18
Building Communication Channels with asyncio
Building Communication Channels with asyncio
Technical requirements
The ecosystem of communication channels
Python example
Client-side communication with aiohttp
Summary
Questions
Further reading
19
Deadlocks
Deadlocks
Technical requirements
The concept of deadlock
Approaches to deadlock situations
The concept of livelock
Summary
Questions
Further reading
20
Starvation
Starvation
Technical requirements
The concept of starvation
The readers-writers problem
Solutions to starvation
Summary
Questions
Further reading
21
Race Conditions
Race Conditions
Technical requirements
The concept of race conditions
Simulating race conditions in Python
Locks as a solution to race conditions
Race conditions in real life
Summary
Questions
Further reading
22
The Global Interpreter Lock
The Global Interpreter Lock
Technical requirements
An introduction to the Global Interpreter Lock
The potential removal of the GIL from Python
How to work with the GIL
Summary
Questions
Further reading
23
The Factory Pattern
The Factory Pattern
The factory method
The abstract factory
Summary
24
The Builder Pattern
The Builder Pattern
Real-world examples
Use cases
Implementation
Summary
25
Other Creational Patterns
Other Creational Patterns
The prototype pattern
Singleton
Summary
26
The Adapter Pattern
The Adapter Pattern
Real-world examples
Use cases
Implementation
Summary
27
The Decorator Pattern
The Decorator Pattern
Real-world examples
Use cases
Implementation
Summary
28
The Bridge Pattern
The Bridge Pattern
Real-world examples
Use cases
Implementation
Summary
29
The Facade Pattern
The Facade Pattern
Real-world examples
Use cases
Implementation
Summary
30
Other Structural Patterns
Other Structural Patterns
The flyweight pattern
The model-view-controller pattern
The proxy pattern
Summary
31
The Chain of Responsibility Pattern
The Chain of Responsibility Pattern
Real-world examples
Use cases
Implementation
Summary
32
The Command Pattern
The Command Pattern
Real-world examples
Use cases
Implementation
Summary
33
The Observer Pattern
The Observer Pattern
Real-world examples
Use cases
Implementation
Summary
Appendix
Appendix
Chapter 8
Chapter 9
Chapter 10
Chapter 11
Chapter 12
Chapter 13
Chapter 14
Chapter 15
Chapter 16
Chapter 17
Chapter 18
Chapter 19
Chapter 20
Chapter 21
Chapter 22
Other Books You May Enjoy
Other Books You May Enjoy
Leave a Review - Let Other Readers Know What You Think
Index
Index

Chapter 7. Parallel Processing

With parallel processing by using multiple cores, you can increase the amount of calculations your program can do in a given time frame without needing a faster processor. The main idea is to divide a problem into independent subunits and use multiple cores to solve those subunits in parallel.

Parallel processing is necessary to tackle large-scale problems. Companies produce massive quantities of data every day that need to be stored in multiple computers and analyzed. Scientists and engineers run parallel code on supercomputers to simulate massive systems.

Parallel processing allows you to take advantage of multicore CPUs as well as GPUs that work extremely well with highly parallel problems. In this chapter, we will cover the following topics:

  • A brief introduction to the fundamentals of parallel processing
  • Illustrating how to parallelize simple problems with the multiprocessing Python library
  • Using the simple ProcessPoolExecutor interface
  • Parallelizing our programs...
Previous Section
End of Section 1
Next Section