Creating basic logs in Airflow
The internal Airflow logging library is based on the Python built-in logs, which provide flexible and configurable forms to capture and store log messages using different components of directed acyclic graphs (DAGs). Let’s start this chapter by covering the basic concepts of how Airflow logs work. This knowledge will allow us to apply more advanced concepts and create mature data ingestion pipelines in real-life projects.
In this recipe, we will create a simple DAG to generate logs based on the default configurations of Airflow. We will also understand how Airflow internally sets the logging architecture.
Getting ready
Refer to the Technical requirements section for this recipe, since we will handle it with the same technology.
Since we will create a new DAG, let’s create a folder under the dag/ directory called basic_logging and a file inside it called basic_logging_dag.py to insert our script. By the end, your folder...