Apache Flink supports various connectors that allow data read/writes across various technologies. Let's learn more about this.
Kafka is a publish-subscribe, distributed, message queuing system that allows users to publish messages to a certain topic; this is then distributed to the subscribers of the topic. Flink provides options to define a Kafka consumer as a data source in Flink Streaming. In order to use the Flink Kafka connector, we need to use a specific JAR file.
The following diagram shows how the Flink Kafka connector works:
We need to use the following Maven dependency to use the connector. I have been using Kafka version 0.9 so I will be adding the following dependency in pom.xml
:
<dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-connector-kafka-0.9_2.11/artifactId> <version>1.1.4</version> </dependency>
Now let's try to understand how to use the Kafka consumer as the Kafka source:
In...