This section will explain how you can read the server log from a Kafka topic. We will use the Kafka spout integration available on GitHub at https://github.com/wurstmeister/storm-kafka-0.8-plus for consuming the data from Kafka. This section also defines the LogProcessingTopology
topology that will chain together all the bolts created in the preceding sections. Let's perform the following steps to consume the data from Kafka and define a topology:
Add the following dependency and repository for Kafka in
pom.xml
:<dependency> <groupId>net.wurstmeister.storm</groupId> <artifactId>storm-kafka-0.8-plus</artifactId> <version>0.4.0</version> </dependency>
Add the following build plugins to
pom.xml
. These plugins will let us executeLogProcessingTopology
using Maven:<build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId>...