Book Image

Apache Kafka Cookbook

By : Saurabh Minni
Book Image

Apache Kafka Cookbook

By: Saurabh Minni

Overview of this book

<p>This book will give you details about how to manage and administer your Apache Kafka Cluster.</p> <p>We will cover topics like how to configure your broker, producer, and consumer for maximum efficiency for your situation. Also, you will learn how to maintain and administer your cluster for fault tolerance. We will also explore tools provided with Apache Kafka to do regular maintenance operations. We shall also look at how to easily integrate Apache Kafka with big data tools like Hadoop, Apache Spark, Apache Storm, and Elasticsearch.</p>

Related Content you might be interested in

Current Title:

Small book image
Apache Kafka Cookbook
Saurabh Minni
Dec, 2015 | 128 pages
No titles found
Table of Contents (15 chapters)
Apache Kafka Cookbook
Apache Kafka Cookbook
Credits
Credits
About the Author
About the Author
About the Reviewers
About the Reviewers
www.PacktPub.com
www.PacktPub.com
Preface
Preface
Free Chapter
1
Initiating Kafka
Initiating Kafka
Introduction
Setting up multiple Kafka brokers
Creating topics
Sending some messages from the console
Consuming from the console
2
Configuring Brokers
Configuring Brokers
Introduction
Configuring the basic settings
Configuring threads and performance
Configuring the log settings
Configuring the replica settings
Configuring the ZooKeeper settings
Configuring other miscellaneous parameters
3
Configuring a Producer and Consumer
Configuring a Producer and Consumer
Introduction
Configuring the basic settings for producer
Configuring the thread and performance for producer
Configuring the basic settings for consumer
Configuring the thread and performance for consumer
Configuring the log settings for consumer
Configuring the ZooKeeper settings for consumer
Other configurations for consumer
4
Managing Kafka
Managing Kafka
Introduction
Consumer offset checker
Understanding dump log segments
Exporting the ZooKeeper offsets
Importing the ZooKeeper offsets
Using GetOffsetShell
Using the JMX tool
Using the Kafka migration tool
The MirrorMaker tool
Replay Log Producer
Simple Consumer Shell
State Change Log Merger
Updating offsets in Zookeeper
Verifying consumer rebalance
5
Integrating Kafka with Java
Integrating Kafka with Java
Introduction
Writing a simple producer
Writing a simple consumer
Writing a high-level consumer
Writing a producer with message partitioning
Multithreaded consumers in Kafka
6
Operating Kafka
Operating Kafka
Introduction
Adding and removing topics
Modifying topics
Implementing a graceful shutdown
Balancing leadership
Mirroring data between Kafka clusters
Expanding clusters
Increasing the replication factor
Checking the consumer position
Decommissioning brokers
7
Integrating Kafka with Third-Party Platforms
Integrating Kafka with Third-Party Platforms
Introduction
Using Flume
Using Gobblin
Using Logstash
Configuring Kafka for real-time
Integrating Spark with Kafka
Integrating Storm with Kafka
Integrating Elasticsearch with Kafka
Integrating SolrCloud with Kafka
8
Monitoring Kafka
Monitoring Kafka
Introduction
Monitoring server stats
Monitoring producer stats
Index
Index

About the Author

Saurabh Minni has an BE in computer science and engineering. A polyglot programmer with over 10 years of experience, he has worked on a variety of technologies, including Assembly, C, C++, Java, Delphi, JavaScript, Android, iOS, PHP, Python, ZMQ, Redis, Mongo, Kyoto Tycoon, Cocoa, Carbon, Apache Storm, and Elasticsearch. In short, he is a programmer at heart, and loves learning new tech-related things each day.

Currently, he is working as a technical architect with Near (an amazing start-up that builds a location intelligence platform). Apart from handling several projects, he was also responsible for deploying Apache Kafka cluster. This was instrumental in streamlining the consumption of data in the big data processing systems, such as Apache Storm, Hadoop, and others at Near.

He has also reviewed Learning Apache Kafka, Packt Publishing.

He is reachable on Twitter at @the100rabh and on Github at https://github.com/the100rabh/.

This book would not have been possible without the continuous support of my parents, Suresh and Sarla, and my wife, Puja. Thank you for always being there.

I would also like to thank Arun Vijayan, chief architect at Near, who encouraged me to learn and experiment with different technologies at work. Without his encouragement, this book would not exist.

Previous Section
Next Chapter