Summary
In this chapter, we covered an introduction to data processing on AWS, specifically focusing on ML and data science. We looked at how data processing for ML is unique and why it is such a critical and significant component of the overall ML workflow. We went through some of the challenges when dealing with large and distributed datasets and data sources and how to work with these at scale. We discussed the importance of having a reliable and repeatable data processing workflow for ML. We then covered some of the key capabilities that are needed in tooling and the frameworks used for data processing for ML, which include the ability to detect bias present in real-world data, the ability to detect and fix data imbalances, the ability to perform quick and error-free transformations and run preprocessing reports and visualizations at scale, as well as the ability to ingest data at scale.
As enterprises move from experimentation and research to production, the focus switches...