Book Image

Automated Machine Learning on AWS

By : Trenton Potgieter
Book Image

Automated Machine Learning on AWS

By: Trenton Potgieter

Overview of this book

AWS provides a wide range of solutions to help automate a machine learning workflow with just a few lines of code. With this practical book, you'll learn how to automate a machine learning pipeline using the various AWS services. Automated Machine Learning on AWS begins with a quick overview of what the machine learning pipeline/process looks like and highlights the typical challenges that you may face when building a pipeline. Throughout the book, you'll become well versed with various AWS solutions such as Amazon SageMaker Autopilot, AutoGluon, and AWS Step Functions to automate an end-to-end ML process with the help of hands-on examples. The book will show you how to build, monitor, and execute a CI/CD pipeline for the ML process and how the various CI/CD services within AWS can be applied to a use case with the Cloud Development Kit (CDK). You'll understand what a data-centric ML process is by working with the Amazon Managed Services for Apache Airflow and then build a managed Airflow environment. You'll also cover the key success criteria for an MLSDLC implementation and the process of creating a self-mutating CI/CD pipeline using AWS CDK from the perspective of the platform engineering team. By the end of this AWS book, you'll be able to effectively automate a complete machine learning pipeline and deploy it to production.
Table of Contents (18 chapters)
1
Section 1: Fundamentals of the Automated Machine Learning Process and AutoML on AWS
5
Section 2: Automating the Machine Learning Process with Continuous Integration and Continuous Delivery (CI/CD)
8
Section 3: Optimizing a Source Code-Centric Approach to Automated Machine Learning
11
Section 4: Optimizing a Data-Centric Approach to Automated Machine Learning
14
Section 5: Automating the End-to-End Production Application on AWS

Using Airflow to process the abalone dataset

To set the scene, you will recall from Chapter 1, Getting Started with Automated Machine Learning on AWS, that the ACME Fishing Logistics company uses an outdated dataset, found in the UCI Machine Learning Repository, to train the ML model. The ML practitioners have found that since an ML model is only as good as the data it's trained on, they can tweak and tune the model as much as they want, but without newer data, the production model can't be improved upon.

To resolve this problem, ACME has hired an external company to survey abalone catches and supply daily updates of the surveyed dataset. This means that the already tuned ML model can be retrained on fresh data, and thus be further optimized. This also means that the data engineering teams need to orchestrate a process, or data pipeline, to merge the original dataset with the new survey data and supply the new training, validation, and testing dataset to a new model training...