9.1 Description
We need to build a data validating, cleaning, and standardizing application. A data inspection notebook is a handy starting point for this design work. The goal is a fully-automated application to reflect the lessons learned from inspecting the data.
A data preparation pipeline has the following conceptual tasks:
Validate the acquired source text to be sure it’s usable and to mark invalid data for remediation.
Clean any invalid raw data where necessary; this expands the available data in those cases where sensible cleaning can be defined.
Convert the validated and cleaned source data from text (or bytes) to usable Python objects.
Where necessary, standardize the code or ranges of source data. The requirements here vary with the problem domain.
The goal is to create clean, standardized data for subsequent analysis. Surprises occur all the time. There are several sources:
Technical problems with file formats of the upstream software. The intent of the acquisition...