Apache Kafka Cookbook

Book Image

Apache Kafka Cookbook

By : Saurabh Minni

Book Image

Apache Kafka Cookbook

By: Saurabh Minni

Overview of this book

<p>This book will give you details about how to manage and administer your Apache Kafka Cluster.</p> <p>We will cover topics like how to configure your broker, producer, and consumer for maximum efficiency for your situation. Also, you will learn how to maintain and administer your cluster for fault tolerance. We will also explore tools provided with Apache Kafka to do regular maintenance operations. We shall also look at how to easily integrate Apache Kafka with big data tools like Hadoop, Apache Spark, Apache Storm, and Elasticsearch.</p>

Apache Kafka Cookbook

Apache Kafka Cookbook

Credits

About the Author

About the Author

About the Reviewers

About the Reviewers

www.PacktPub.com

www.PacktPub.com

Preface

Free Chapter

Initiating Kafka

Initiating Kafka

Setting up multiple Kafka brokers

Creating topics

Sending some messages from the console

Consuming from the console

Configuring Brokers

Configuring Brokers

Configuring the basic settings

Configuring threads and performance

Configuring the log settings

Configuring the replica settings

Configuring the ZooKeeper settings

Configuring other miscellaneous parameters

Configuring a Producer and Consumer

Configuring a Producer and Consumer

Configuring the basic settings for producer

Configuring the thread and performance for producer

Configuring the basic settings for consumer

Configuring the thread and performance for consumer

Configuring the log settings for consumer

Configuring the ZooKeeper settings for consumer

Other configurations for consumer

Managing Kafka

Consumer offset checker

Understanding dump log segments

Exporting the ZooKeeper offsets

Importing the ZooKeeper offsets

Using GetOffsetShell

Using the JMX tool

Using the Kafka migration tool

The MirrorMaker tool

Replay Log Producer

Simple Consumer Shell

State Change Log Merger

Updating offsets in Zookeeper

Verifying consumer rebalance

Integrating Kafka with Java

Integrating Kafka with Java

Writing a simple producer

Writing a simple consumer

Writing a high-level consumer

Writing a producer with message partitioning

Multithreaded consumers in Kafka

Operating Kafka

Operating Kafka

Adding and removing topics

Modifying topics

Implementing a graceful shutdown

Balancing leadership

Mirroring data between Kafka clusters

Expanding clusters

Increasing the replication factor

Checking the consumer position

Decommissioning brokers

Integrating Kafka with Third-Party Platforms

Integrating Kafka with Third-Party Platforms

Configuring Kafka for real-time

Integrating Spark with Kafka

Integrating Storm with Kafka

Integrating Elasticsearch with Kafka

Integrating SolrCloud with Kafka

Monitoring Kafka

Monitoring Kafka

Monitoring server stats

Monitoring producer stats

Index

Customer Reviews

5 star

0

4 star

0

3 star

0

2 star

0

1 star

0

Configuring other miscellaneous parameters

Besides the earlier mentioned configurations, there are some other configurations that also need to be set.

Getting ready

Just open the server.properties file in your favorite editor.

How to do it...

We will look at the default values of the properties as follows:

Set auto.create.topics.enable:
```
auto.create.topics.enable=true
```
Set controlled.shutdown.enable:
```
controlled.shutdown.enable=true
```
Set controlled.shutdown.max.retries:
```
controlled.shutdown.max.retries=3
```

Set controlled.shutdown.retry.backoff.ms:

controlled.shutdown.retry.backoff.ms=5000

Set auto.leader.rebalance.enable:
```
auto.leader.rebalance.enable=true
```

Set leader.imbalance.per.broker.percentage:

leader.imbalance.per.broker.percentage=10

Set leader.imbalance.check.interval.seconds:

leader.imbalance.check.interval.seconds=300

Set offset.metadata.max.bytes:
```
offset.metadata.max.bytes=4096
```
Set max.connections.per.ip:
```
max.connections.per.ip=Int.MaxValue
```
Set connections.max.idle.ms:
```
connections.max.idle...
```