Book Image

Apache Kafka 1.0 Cookbook

By : Alexey Zinoviev, Raúl Estrada
Book Image

Apache Kafka 1.0 Cookbook

By: Alexey Zinoviev, Raúl Estrada

Overview of this book

Apache Kafka provides a unified, high-throughput, low-latency platform to handle real-time data feeds. This book will show you how to use Kafka efficiently, and contains practical solutions to the common problems that developers and administrators usually face while working with it. This practical guide contains easy-to-follow recipes to help you set up, configure, and use Apache Kafka in the best possible manner. You will use Apache Kafka Consumers and Producers to build effective real-time streaming applications. The book covers the recently released Kafka version 1.0, the Confluent Platform and Kafka Streams. The programming aspect covered in the book will teach you how to perform important tasks such as message validation, enrichment and composition.Recipes focusing on optimizing the performance of your Kafka cluster, and integrate Kafka with a variety of third-party tools such as Apache Hadoop, Apache Spark, and Elasticsearch will help ease your day to day collaboration with Kafka greatly. Finally, we cover tasks related to monitoring and securing your Apache Kafka cluster using tools such as Ganglia and Graphite. If you're looking to become the go-to person in your organization when it comes to working with Apache Kafka, this book is the only resource you need to have.

Related Content you might be interested in

Current Title:

Small book image
Apache Kafka 1.0 Cookbook
Alexey Zinoviev | Raúl Estrada
Dec, 2017 | 250 pages
Small book image
Building Data Streaming Applications with Apache Kafka
Manish Kumar | Chanchal Singh
Aug, 2017 | 278 pages
Small book image
Practical Real-time Data Processing and Analytics
Saurabh Gupta | Shilpi Saxena
Sep, 2017 | 360 pages
Small book image
Mastering Apache Storm
Ankit Jain
Aug, 2017 | 284 pages
Table of Contents (18 chapters)
Title Page
Title Page
Credits
Credits
About the Author
About the Author
About the Reviewers
About the Reviewers
www.PacktPub.com
www.PacktPub.com
Customer Feedback
Customer Feedback
Dedication
Dedication
Preface
Preface
Free Chapter
1
Configuring Kafka
Configuring Kafka
Introduction
Installing Kafka
Running Kafka
Configuring Kafka brokers
Configuring Kafka topics
Creating a message console producer
Creating a message console consumer
Configuring the broker settings
Configuring threads and performance
Configuring the log settings
Configuring the replica settings
Configuring the ZooKeeper settings
Configuring other miscellaneous parameters
2
Kafka Clusters
Kafka Clusters
Introduction
Configuring a single-node single-broker cluster – SNSB
SNSB – creating a topic, producer, and consumer
Configuring a single-node multiple-broker cluster – SNMB
SNMB – creating a topic, producer, and consumer
Configuring a multiple-node multiple-broker cluster – MNMB
3
Message Validation
Message Validation
Introduction
Modeling the events
Setting up the project
Reading from Kafka
Writing to Kafka
Running ProcessingApp
Coding the validator
Running the validator
4
Message Enrichment
Message Enrichment
Introduction
Geolocation extractor
Geolocation enricher
Currency price extractor
Currency price enricher
Running the currency price enricher
Modeling the events
Setting up the project
Open weather extractor
Location temperature enricher
Running the location temperature enricher
5
The Confluent Platform
The Confluent Platform
Introduction
Installing the Confluent Platform
Using Kafka operations
Monitoring with the Confluent Control Center
Using the Schema Registry
Using the Kafka REST Proxy
Using Kafka Connect
6
Kafka Streams
Kafka Streams
Introduction
Setting up the project
Running the streaming application
7
Managing Kafka
Managing Kafka
Introduction
Managing consumer groups
Dumping log segments
Importing ZooKeeper offsets
Using the GetOffsetShell
Using the JMX tool
Using the MirrorMaker tool
Replaying log producer
Using state change log merger
8
Operating Kafka
Operating Kafka
Introduction
Adding or removing topics
Modifying message topics
Implementing a graceful shutdown
Balancing leadership
Expanding clusters
Increasing the replication factor
Decommissioning brokers
Checking the consumer position
9
Monitoring and Security
Monitoring and Security
Introduction
Monitoring server statistics
Monitoring producer statistics
Monitoring consumer statistics
Connecting with the help of Graphite
Monitoring with the help of Ganglia
Implementing authentication using SSL
Implementing authentication using SASL/Kerberos
10
Third-Party Tool Integration
Third-Party Tool Integration
Introduction
Moving data between Kafka nodes with Flume
Writing to an HDFS cluster with Gobblin
Moving data from Kafka to Elastic with Logstash
Connecting Spark streams and Kafka
Ingesting data from Kafka to Storm
Pushing data from Kafka to Elastic
Inserting data from Kafka to SolrCloud
Building a Kafka producer with Akka
Building a Kafka consumer with Akka
Storing data in Cassandra
Running Kafka on Mesos
Reading Kafka with Apache Beam
Writing to Kafka from Apache Beam

Expanding clusters

Adding nodes to an existing cluster is not the same as building a new Kafka cluster. Adding nodes to an existing cluster is easy. We do this by assigning them a unique broker ID, but they are not going to receive data automatically. A cluster reconfiguration is needed to indicate which partition replicas go where. Then, the partitions will move to the newly added nodes. This recipe shows how to do that.

Getting ready

For this recipe, Kafka must be installed, ZooKeeper should be running, and the broker should be running with some topics created on it.

How to do it...

  1. This recipe moves all partitions for existing topics: topic_1 and topic_2. The newly generated brokers are broker_7 and broker_8 (suppose that brokers 1 to 6 already exist). After finishing the movement, all partitions for topic_1 and topic_2 will exist only in broker_7 and broker_8.
  2. The tool only accepts JSON files as input; let's create the JSON file as follows:
$ cat to_reassign.json 
 
{"topics": [{"topic": ...
Previous Section
End of Section 7
Next Section