Book Image

Platform and Model Design for Responsible AI

By : Amita Kapoor, Sharmistha Chatterjee
Book Image

Platform and Model Design for Responsible AI

By: Amita Kapoor, Sharmistha Chatterjee

Overview of this book

AI algorithms are ubiquitous and used for tasks, from recruiting to deciding who will get a loan. With such widespread use of AI in the decision-making process, it’s necessary to build an explainable, responsible, transparent, and trustworthy AI-enabled system. With Platform and Model Design for Responsible AI, you’ll be able to make existing black box models transparent. You’ll be able to identify and eliminate bias in your models, deal with uncertainty arising from both data and model limitations, and provide a responsible AI solution. You’ll start by designing ethical models for traditional and deep learning ML models, as well as deploying them in a sustainable production setup. After that, you’ll learn how to set up data pipelines, validate datasets, and set up component microservices in a secure and private way in any cloud-agnostic framework. You’ll then build a fair and private ML model with proper constraints, tune the hyperparameters, and evaluate the model metrics. By the end of this book, you’ll know the best practices to comply with data privacy and ethics laws, in addition to the techniques needed for data anonymization. You’ll be able to develop models with explainability, store them in feature stores, and handle uncertainty in model predictions.
Table of Contents (21 chapters)
1
Part 1: Risk Assessment Machine Learning Frameworks in a Global Landscape
5
Part 2: Building Blocks and Patterns for a Next-Generation AI Ecosystem
9
Part 3: Design Patterns for Model Optimization and Life Cycle Management
14
Part 4: Implementing an Organization Strategy, Best Practices, and Use Cases

Katib for hyperparameter tuning

Katib is a scalable, Kubernetes-native AutoML platform that facilitates both hyperparameter tuning and NAS. Figure 6.13 shows the design of Katib. To learn more about how it works, readers should refer to Katib: A Distributed General AutoML Platform on Kubernetes:

Figure 6.13 – The design of Katib as a general AutoML system (Figure 2 from the paper: https://www.usenix.org/system/files/opml19papers-zhou.pdf)

Figure 6.13 – The design of Katib as a general AutoML system (Figure 2 from the paper: https://www.usenix.org/system/files/opml19papers-zhou.pdf)

Katib supports hyperparameter adjustment through the command line via a YAML file specification, as well as the Jupyter Notebook and the Python SDK. It also has a graphical UI for specifying tuning settings and visualizing the results, as shown here:

Figure 6.14 – The graphical interface of Katib

Figure 6.14 – The graphical interface of Katib

Katib allows you to choose a measure and whether to reduce or increase it. You can choose the hyperparameters you want to tweak and see the results of the entire experiment and individual...