Book Image

Building Data Streaming Applications with Apache Kafka

By : Chanchal Singh, Manish Kumar

Book Image

Building Data Streaming Applications with Apache Kafka

By: Chanchal Singh, Manish Kumar

Overview of this book

Apache Kafka is a popular distributed streaming platform that acts as a messaging queue or an enterprise messaging system. It lets you publish and subscribe to a stream of records, and process them in a fault-tolerant way as they occur. This book is a comprehensive guide to designing and architecting enterprise-grade streaming applications using Apache Kafka and other big data tools. It includes best practices for building such applications, and tackles some common challenges such as how to use Kafka efficiently and handle high data volumes with ease. This book first takes you through understanding the type messaging system and then provides a thorough introduction to Apache Kafka and its internal details. The second part of the book takes you through designing streaming application using various frameworks and tools such as Apache Spark, Apache Storm, and more. Once you grasp the basics, we will take you through more advanced concepts in Apache Kafka such as capacity planning and security. By the end of this book, you will have all the information you need to be comfortable with using Apache Kafka, and to design efficient streaming data applications with it.

Preface

What this book covers

What you need for this book

Who this book is for

Reader feedback

Customer support

Free Chapter

Introduction to Messaging Systems

Introduction to Messaging Systems

Understanding the principles of messaging systems

Understanding messaging systems

Peeking into a point-to-point messaging system

Publish-subscribe messaging system

Advance Queuing Messaging Protocol

Using messaging systems in big data streaming applications

Introducing Kafka the Distributed Messaging Platform

Introducing Kafka the Distributed Messaging Platform

Kafka's architecture

Message partitions

Replication and replicated logs

Message producers

Message consumers

Role of Zookeeper

Deep Dive into Kafka Producers

Deep Dive into Kafka Producers

Kafka producer internals

Kafka Producer APIs

Java Kafka producer example

Common messaging publishing patterns

Deep Dive into Kafka Consumers

Deep Dive into Kafka Consumers

Kafka consumer internals

Kafka consumer APIs

Java Kafka consumer

Scala Kafka consumer

Common message consuming patterns

Building Spark Streaming Applications with Kafka

Building Spark Streaming Applications with Kafka

Introduction to Spark

Spark Streaming

Use case log processing - fraud IP detection

Building Storm Applications with Kafka

Building Storm Applications with Kafka

Introduction to Apache Storm

Introduction to Apache Heron

Integrating Apache Kafka with Apache Storm - Java

Integrating Apache Kafka with Apache Storm - Scala

Use case – log processing in Storm, Kafka, Hive

Using Kafka with Confluent Platform

Using Kafka with Confluent Platform

Introduction to Confluent Platform

Deep driving into Confluent architecture

Understanding Kafka Connect and Kafka Stream

Playing with Avro using Schema Registry

Moving Kafka data to HDFS

Building ETL Pipelines Using Kafka

Building ETL Pipelines Using Kafka

Considerations for using Kafka in ETL pipelines

Introducing Kafka Connect

Deep dive into Kafka Connect

Introductory examples of using Kafka Connect

Kafka Connect common use cases

Building Streaming Applications Using Kafka Streams

Building Streaming Applications Using Kafka Streams

Introduction to Kafka Streams

Kafka Stream architecture

Integrated framework advantages

Understanding tables and Streams together

Use case example of Kafka Streams

Kafka Cluster Deployment

Kafka Cluster Deployment

Kafka cluster internals

Capacity planning

Single cluster deployment

Multicluster deployment

Decommissioning brokers

Using Kafka in Big Data Applications

Using Kafka in Big Data Applications

Managing high volumes in Kafka

Kafka message delivery semantics

Big data and Kafka common usage patterns

Kafka and data governance

Alerting and monitoring

Useful Kafka matrices

Securing Kafka

An overview of securing Kafka

Wire encryption using SSL

Kerberos SASL for authentication

Understanding ACL and authorization

Understanding Zookeeper authentication

Apache Ranger for authorization

Streaming Application Design Considerations

Streaming Application Design Considerations

Latency and throughput

Data and state persistence

External data lookups

Data serialization

Level of parallelism

Out-of-order events

Message processing semantics

Customer Reviews

5 star

0

4 star

0

3 star

0

2 star

0

1 star

0

Deep dive into Kafka Connect

Let's get into the architecture of Kafka Connect. The following figure gives a good idea of Kafka Connect:

Kafka Connect architecture

Kafka Connect has three major models in its design:

Connector: A Connector is configured by defining the Connector class and configuration. The Connector class is defined based on the source or target of the data, which means that it will be different for the Database source and File source. It is then followed by setting up the configuration for these classes. For example, configuration for the Database source could be the IP of the database, the username and password to connect to the database, and so on. The Connector creates a set of tasks, which is actually responsible for copying data from the source or copying data to the target. Connectors are of two types:
- Source Connector: This is responsible for ingesting...