-
Book Overview & Buying
-
Table Of Contents
Python Automation Cookbook - Third Edition
By :
The processing presented in the previous recipe works well, but it processes each file one by one. When we have a small number of files this may be fine, but when there are huge numbers of files to handle this will not be efficient. For most file processing we typically use a single CPU core, which is not the best approach for this type of number-crunching task. Instead, we can use multiple CPUs to process more than one chunk of data at a time.
In this recipe we will see how to process the files in parallel, making use of all the cores of the computer to speed up the process and greatly increase the throughput.
Remember what we said in the previous recipe about treating data in smaller chunks? In this case, we will chunk the data by location.
We will use the CSV file resulting from the previous recipe that receives and transforms logs in the following format:
[<Timestamp>] - SALE - PRODUCT: <product id> - PRICE: <price>
Each line...