Setting up Airflow for orchestration
To productionize the online and batch model, we need a workflow orchestration tool that can run the ML pipelines for us on schedule. There are a bunch of tools available, such as Apache Airflow, AWS Step Functions, and SageMaker Pipelines. You can also run it as GitHub workflows if you prefer. Depending on the tools you are familiar with or offered at your organization, orchestration may differ. For this exercise, we will use Amazon Managed Workflows for Apache Airflow (MWAA). As the name suggests, it is an Apache Airflow-managed service by AWS. Let's create an Amazon MWAA environment in AWS.
Important Note
Amazon MWAA doesn't have a free trial. You can view the pricing for the usage at this URL: https://aws.amazon.com/managed-workflows-for-apache-airflow/pricing/. Alternatively, you can choose to run Airflow locally or on EC2 instances (EC2 has free tier resources). You can find the setup instructions to run Airflow locally or on...