Book Image

Building Data Streaming Applications with Apache Kafka

By : Chanchal Singh, Manish Kumar

Book Image

Building Data Streaming Applications with Apache Kafka

By: Chanchal Singh, Manish Kumar

Overview of this book

Apache Kafka is a popular distributed streaming platform that acts as a messaging queue or an enterprise messaging system. It lets you publish and subscribe to a stream of records, and process them in a fault-tolerant way as they occur. This book is a comprehensive guide to designing and architecting enterprise-grade streaming applications using Apache Kafka and other big data tools. It includes best practices for building such applications, and tackles some common challenges such as how to use Kafka efficiently and handle high data volumes with ease. This book first takes you through understanding the type messaging system and then provides a thorough introduction to Apache Kafka and its internal details. The second part of the book takes you through designing streaming application using various frameworks and tools such as Apache Spark, Apache Storm, and more. Once you grasp the basics, we will take you through more advanced concepts in Apache Kafka such as capacity planning and security. By the end of this book, you will have all the information you need to be comfortable with using Apache Kafka, and to design efficient streaming data applications with it.

Preface

What this book covers

What you need for this book

Who this book is for

Reader feedback

Customer support

Free Chapter

Introduction to Messaging Systems

Introduction to Messaging Systems

Understanding the principles of messaging systems

Understanding messaging systems

Peeking into a point-to-point messaging system

Publish-subscribe messaging system

Advance Queuing Messaging Protocol

Using messaging systems in big data streaming applications

Introducing Kafka the Distributed Messaging Platform

Introducing Kafka the Distributed Messaging Platform

Kafka's architecture

Message partitions

Replication and replicated logs

Message producers

Message consumers

Role of Zookeeper

Deep Dive into Kafka Producers

Deep Dive into Kafka Producers

Kafka producer internals

Kafka Producer APIs

Java Kafka producer example

Common messaging publishing patterns

Deep Dive into Kafka Consumers

Deep Dive into Kafka Consumers

Kafka consumer internals

Kafka consumer APIs

Java Kafka consumer

Scala Kafka consumer

Common message consuming patterns

Building Spark Streaming Applications with Kafka

Building Spark Streaming Applications with Kafka

Introduction to Spark

Spark Streaming

Use case log processing - fraud IP detection

Building Storm Applications with Kafka

Building Storm Applications with Kafka

Introduction to Apache Storm

Introduction to Apache Heron

Integrating Apache Kafka with Apache Storm - Java

Integrating Apache Kafka with Apache Storm - Scala

Use case – log processing in Storm, Kafka, Hive

Using Kafka with Confluent Platform

Using Kafka with Confluent Platform

Introduction to Confluent Platform

Deep driving into Confluent architecture

Understanding Kafka Connect and Kafka Stream

Playing with Avro using Schema Registry

Moving Kafka data to HDFS

Building ETL Pipelines Using Kafka

Building ETL Pipelines Using Kafka

Considerations for using Kafka in ETL pipelines

Introducing Kafka Connect

Deep dive into Kafka Connect

Introductory examples of using Kafka Connect

Kafka Connect common use cases

Building Streaming Applications Using Kafka Streams

Building Streaming Applications Using Kafka Streams

Introduction to Kafka Streams

Kafka Stream architecture

Integrated framework advantages

Understanding tables and Streams together

Use case example of Kafka Streams

Kafka Cluster Deployment

Kafka Cluster Deployment

Kafka cluster internals

Capacity planning

Single cluster deployment

Multicluster deployment

Decommissioning brokers

Using Kafka in Big Data Applications

Using Kafka in Big Data Applications

Managing high volumes in Kafka

Kafka message delivery semantics

Big data and Kafka common usage patterns

Kafka and data governance

Alerting and monitoring

Useful Kafka matrices

Securing Kafka

An overview of securing Kafka

Wire encryption using SSL

Kerberos SASL for authentication

Understanding ACL and authorization

Understanding Zookeeper authentication

Apache Ranger for authorization

Streaming Application Design Considerations

Streaming Application Design Considerations

Latency and throughput

Data and state persistence

External data lookups

Data serialization

Level of parallelism

Out-of-order events

Message processing semantics

Customer Reviews

5 star

0

4 star

0

3 star

0

2 star

0

1 star

0

Kafka Connect common use cases

You have learned about Kafka Connect in detail. We know Kafka Connect is used for copying data in and out of Kafka.

Let's understand a few common use cases of Kafka Connect:

Copying data to HDFS: User wants to copy data from Kafka topics to HDFS for various reasons. A few want to copy it to HDFS just to take a backup of the history data, others may want to copy it to HDFS for batch processing. However, there are already many open source tools available, such as Camus, Gobblin, Flume, and so on, but maintaining, installing, and running these jobs takes more effort than what Kafka Connect provides. Kafka Connect copies data from topics in parallel and is capable of scaling up more if required.
Replication: Replicating Kafka topics from one cluster to another cluster is also a popular feature offered by Kafka Connect. You may want to replicate...