Book Overview & Buying
Table Of Contents
Feedback & Rating

Building Data Streaming Applications with Apache Kafka

By : Singh, Kumar

3.8 (4)

Buy this Book

Building Data Streaming Applications with Apache Kafka

3.8 (4)

By: Singh, Kumar

Buy this Book

Overview of this book

Apache Kafka is a popular distributed streaming platform that acts as a messaging queue or an enterprise messaging system. It lets you publish and subscribe to a stream of records, and process them in a fault-tolerant way as they occur. This book is a comprehensive guide to designing and architecting enterprise-grade streaming applications using Apache Kafka and other big data tools. It includes best practices for building such applications, and tackles some common challenges such as how to use Kafka efficiently and handle high data volumes with ease. This book first takes you through understanding the type messaging system and then provides a thorough introduction to Apache Kafka and its internal details. The second part of the book takes you through designing streaming application using various frameworks and tools such as Apache Spark, Apache Storm, and more. Once you grasp the basics, we will take you through more advanced concepts in Apache Kafka such as capacity planning and security. By the end of this book, you will have all the information you need to be comfortable with using Apache Kafka, and to design efficient streaming data applications with it.

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Free Chapter

Introduction to Messaging Systems

Understanding the principles of messaging systems

Understanding messaging systems

Peeking into a point-to-point messaging system

Publish-subscribe messaging system

Advance Queuing Messaging Protocol

Using messaging systems in big data streaming applications

Summary

Introducing Kafka the Distributed Messaging Platform

Kafka origins

Kafka's architecture

Message topics

Message partitions

Replication and replicated logs

Message producers

Message consumers

Role of Zookeeper

Summary

Deep Dive into Kafka Producers

Kafka producer internals

Kafka Producer APIs

Java Kafka producer example

Common messaging publishing patterns

Best practices

Summary

Deep Dive into Kafka Consumers

Kafka consumer internals

Kafka consumer APIs

Java Kafka consumer

Scala Kafka consumer

Common message consuming patterns

Best practices

Summary

Building Spark Streaming Applications with Kafka

Introduction to Spark

Spark Streaming

Use case log processing - fraud IP detection

Producer

Summary

Building Storm Applications with Kafka

Introduction to Apache Storm

Introduction to Apache Heron

Integrating Apache Kafka with Apache Storm - Java

Integrating Apache Kafka with Apache Storm - Scala

Use case – log processing in Storm, Kafka, Hive

Summary

Using Kafka with Confluent Platform

Introduction to Confluent Platform

Deep driving into Confluent architecture

Understanding Kafka Connect and Kafka Stream

Playing with Avro using Schema Registry

Moving Kafka data to HDFS

Summary

Building ETL Pipelines Using Kafka

Considerations for using Kafka in ETL pipelines

Introducing Kafka Connect

Deep dive into Kafka Connect

Introductory examples of using Kafka Connect

Kafka Connect common use cases

Summary

Building Streaming Applications Using Kafka Streams

Introduction to Kafka Streams

Kafka Stream architecture

Integrated framework advantages

Understanding tables and Streams together

Use case example of Kafka Streams

Summary

Kafka Cluster Deployment

Kafka cluster internals

Capacity planning

Single cluster deployment

Multicluster deployment

Decommissioning brokers

Data migration

Summary

Using Kafka in Big Data Applications

Managing high volumes in Kafka

Kafka message delivery semantics

Big data and Kafka common usage patterns

Kafka and data governance

Alerting and monitoring

Useful Kafka matrices

Summary

Securing Kafka

An overview of securing Kafka

Wire encryption using SSL

Kerberos SASL for authentication

Understanding ACL and authorization

Understanding Zookeeper authentication

Apache Ranger for authorization

Best practices

Summary

Streaming Application Design Considerations

Latency and throughput

Data and state persistence

Data sources

External data lookups

Data formats

Data serialization

Level of parallelism

Out-of-order events

Message processing semantics

Summary

Customer Reviews

3.8 (4)

5 star

25%

4 star

50%

3 star

2 star

25%

1 star

Introducing Kafka the Distributed Messaging Platform

In this chapter, we will introduce Kafka, a widely adopted scalable, performant, and distributed messaging platform. We will touch-base on different Kafka components and how they work in coherence to achieve reliable message delivery. You should see this chapter as a foundation chapter on Kafka that will help you establish familiarity with the systems involved. This chapter will help you better grasp the next chapters, which cover various Kafka components in detail. At the end of this chapter, you will have a clear understanding of Kafka's architecture and fundamental components of the Kafka messaging system.

We will cover the following topics in this chapter: