Book Image

Apache Kafka Cookbook

By : Saurabh Minni
Book Image

Apache Kafka Cookbook

By: Saurabh Minni

Overview of this book

<p>This book will give you details about how to manage and administer your Apache Kafka Cluster.</p> <p>We will cover topics like how to configure your broker, producer, and consumer for maximum efficiency for your situation. Also, you will learn how to maintain and administer your cluster for fault tolerance. We will also explore tools provided with Apache Kafka to do regular maintenance operations. We shall also look at how to easily integrate Apache Kafka with big data tools like Hadoop, Apache Spark, Apache Storm, and Elasticsearch.</p>

Related Content you might be interested in

Current Title:

Small book image
Apache Kafka Cookbook
Saurabh Minni
Dec, 2015 | 128 pages
No titles found
Table of Contents (15 chapters)
Apache Kafka Cookbook
Apache Kafka Cookbook
Credits
Credits
About the Author
About the Author
About the Reviewers
About the Reviewers
www.PacktPub.com
www.PacktPub.com
Preface
Preface
Free Chapter
1
Initiating Kafka
Initiating Kafka
Introduction
Setting up multiple Kafka brokers
Creating topics
Sending some messages from the console
Consuming from the console
2
Configuring Brokers
Configuring Brokers
Introduction
Configuring the basic settings
Configuring threads and performance
Configuring the log settings
Configuring the replica settings
Configuring the ZooKeeper settings
Configuring other miscellaneous parameters
3
Configuring a Producer and Consumer
Configuring a Producer and Consumer
Introduction
Configuring the basic settings for producer
Configuring the thread and performance for producer
Configuring the basic settings for consumer
Configuring the thread and performance for consumer
Configuring the log settings for consumer
Configuring the ZooKeeper settings for consumer
Other configurations for consumer
4
Managing Kafka
Managing Kafka
Introduction
Consumer offset checker
Understanding dump log segments
Exporting the ZooKeeper offsets
Importing the ZooKeeper offsets
Using GetOffsetShell
Using the JMX tool
Using the Kafka migration tool
The MirrorMaker tool
Replay Log Producer
Simple Consumer Shell
State Change Log Merger
Updating offsets in Zookeeper
Verifying consumer rebalance
5
Integrating Kafka with Java
Integrating Kafka with Java
Introduction
Writing a simple producer
Writing a simple consumer
Writing a high-level consumer
Writing a producer with message partitioning
Multithreaded consumers in Kafka
6
Operating Kafka
Operating Kafka
Introduction
Adding and removing topics
Modifying topics
Implementing a graceful shutdown
Balancing leadership
Mirroring data between Kafka clusters
Expanding clusters
Increasing the replication factor
Checking the consumer position
Decommissioning brokers
7
Integrating Kafka with Third-Party Platforms
Integrating Kafka with Third-Party Platforms
Introduction
Using Flume
Using Gobblin
Using Logstash
Configuring Kafka for real-time
Integrating Spark with Kafka
Integrating Storm with Kafka
Integrating Elasticsearch with Kafka
Integrating SolrCloud with Kafka
8
Monitoring Kafka
Monitoring Kafka
Introduction
Monitoring server stats
Monitoring producer stats
Index
Index

Introduction

This chapter explains the configurations of a Kafka broker. Before we get started with Kafka, it is critical to configure it to suit us best. The best part of Kafka is that it is highly configurable. Though most of the time you will be good to go with the default settings, while dealing with scale and performance, you might want to dirty your hands with the configuration that suits your application best.

Previous Section
End of Section 2
Next Section