Book Image

Apache Kafka Cookbook

By : Saurabh Minni
Book Image

Apache Kafka Cookbook

By: Saurabh Minni

Overview of this book

<p>This book will give you details about how to manage and administer your Apache Kafka Cluster.</p> <p>We will cover topics like how to configure your broker, producer, and consumer for maximum efficiency for your situation. Also, you will learn how to maintain and administer your cluster for fault tolerance. We will also explore tools provided with Apache Kafka to do regular maintenance operations. We shall also look at how to easily integrate Apache Kafka with big data tools like Hadoop, Apache Spark, Apache Storm, and Elasticsearch.</p>

Related Content you might be interested in

Current Title:

Small book image
Apache Kafka Cookbook
Saurabh Minni
Dec, 2015 | 128 pages
No titles found
Table of Contents (15 chapters)
Apache Kafka Cookbook
Apache Kafka Cookbook
Credits
Credits
About the Author
About the Author
About the Reviewers
About the Reviewers
www.PacktPub.com
www.PacktPub.com
Preface
Preface
Free Chapter
1
Initiating Kafka
Initiating Kafka
Introduction
Setting up multiple Kafka brokers
Creating topics
Sending some messages from the console
Consuming from the console
2
Configuring Brokers
Configuring Brokers
Introduction
Configuring the basic settings
Configuring threads and performance
Configuring the log settings
Configuring the replica settings
Configuring the ZooKeeper settings
Configuring other miscellaneous parameters
3
Configuring a Producer and Consumer
Configuring a Producer and Consumer
Introduction
Configuring the basic settings for producer
Configuring the thread and performance for producer
Configuring the basic settings for consumer
Configuring the thread and performance for consumer
Configuring the log settings for consumer
Configuring the ZooKeeper settings for consumer
Other configurations for consumer
4
Managing Kafka
Managing Kafka
Introduction
Consumer offset checker
Understanding dump log segments
Exporting the ZooKeeper offsets
Importing the ZooKeeper offsets
Using GetOffsetShell
Using the JMX tool
Using the Kafka migration tool
The MirrorMaker tool
Replay Log Producer
Simple Consumer Shell
State Change Log Merger
Updating offsets in Zookeeper
Verifying consumer rebalance
5
Integrating Kafka with Java
Integrating Kafka with Java
Introduction
Writing a simple producer
Writing a simple consumer
Writing a high-level consumer
Writing a producer with message partitioning
Multithreaded consumers in Kafka
6
Operating Kafka
Operating Kafka
Introduction
Adding and removing topics
Modifying topics
Implementing a graceful shutdown
Balancing leadership
Mirroring data between Kafka clusters
Expanding clusters
Increasing the replication factor
Checking the consumer position
Decommissioning brokers
7
Integrating Kafka with Third-Party Platforms
Integrating Kafka with Third-Party Platforms
Introduction
Using Flume
Using Gobblin
Using Logstash
Configuring Kafka for real-time
Integrating Spark with Kafka
Integrating Storm with Kafka
Integrating Elasticsearch with Kafka
Integrating SolrCloud with Kafka
8
Monitoring Kafka
Monitoring Kafka
Introduction
Monitoring server stats
Monitoring producer stats
Index
Index

Verifying consumer rebalance

After the rebalancing operation, each partition must have selected an owner. One way to verify that is by reading the data from ZooKeeper at /consumers/[consumer_group]/owners/[topic]/[broker_id-partition_id] and finding an entry for each of /brokers/topics/[topic]/[broker-id]. This tool will make your life easier to make sure there is an owner for every partition.

Getting ready

You have a number of nodes in your Kafka node up and running. You also have a consumer group created and running.

How to do it...

  1. From the Kafka folder, run the following command:

    $ bin/kafka-run-class.sh kafka.tools.VerifyConsumerRebalance --zookeeper.connect localhost:2181 --group mytestconsumer

How it works…

The preceding command takes the following arguments:

  • --group: This is used to specify the consumer group.

  • --help: This prints the help message.

  • --zookeeper.connect: This is used to specify the ZooKeeper connect string.

Previous Section
End of Chapter 4
Next Chapter