In this chapter, we have learned all the popular data ingestion tools used in production environments. Sqoop is mainly used to import and export data in and out of RDBMS data stores. Apache Flume is used in real-time systems to import data, mainly from files sources. It supports a wide variety of sources and sinks. Apache NiFi is a fairly new tool and getting very popular these days. It also supports GUI-based ETL development. Hortonworks has started supporting this tool since their HDP 2.4 release. Apache Kafka Connect is another popular tool in the market. It is also a part of the Confluent Data Platform. Kafka Connect can ingest entire databases or collect metrics from all your application servers into Kafka topics, making the data available for stream processing with low latency.
Since we so far know how to build Hadoop clusters and how to ingest data in them, we will...