-
Book Overview & Buying
-
Table Of Contents
Python Automation Cookbook - Third Edition
By :
Some automated tasks will require dealing with large amounts of data. As data grows, two new and distinct problems appear: processing takes too long and input data quality issues cause more problems.
Both problems are well known in the realm of data science when dealing with large quantities of data, but the problems can appear even at a smaller scale.
The quality of input data is highly related to the number of sources of the data. In general, data from a single source will be more consistent, but using a single source is limiting. And even if the data comes from the same source, it can still contain inconsistencies or errors.
Examples of differences include regional differences, such as different date formats or currencies, extra information, different names for the same concept (including spelling differences), typos, generally poor data quality including errors… The list is long!
To compare apples with apples, the input data will probably need...