As explained in Chapter 4, Storm and Kafka Integration, Kafka is a distributed messaging queue and can integrate with Storm very well. In this section, you'll see how to write a Kafka producer that will read the server logfile and produce the log in Kafka.

As we all know, Storm provides guaranteed message processing, which means every message that enters the Storm topology will be processed at least once. In Storm, data loss is possible only at the spout. This happens if the processing capacity of the Storm spout is less than the producing capacity of the data publisher. Hence, to avoid data loss at the Storm spout, we will generally publish the data into a messaging queue, and the Storm spout will use that messaging queue as the data source.

We will create a Maven project that will publish the server log into a Kafka broker. Perform the following steps to create the server log producer: