Handling databases with NiFi processors
In the previous sections, you learned how to read and write CSV and JSON files using Python. Reading files is such a common task that tools such as NiFi have prebuilt processors to handle it. In this section, you will build the same data pipeline as in the previous section. In NiFi, the data pipeline will look as shown in the following screenshot:
Figure 4.9 – A NiFi data pipeline to move data from PostgreSQL to Elasticsearch
The data pipeline contains one more task than the Airflow version, but otherwise, it should look straightforward. The following sections will walk you through building the data pipeline.
Extracting data from PostgreSQL
The processor most used for handling relational databases in NiFi is the ExecuteSQLRecord processor. Drag the Processor icon to the canvas, and search for the ExecuteSQLRecord processor. Once it has been added to the canvas, you need to configure it.