Sign In Start Free Trial

Book Overview & Buying
Table Of Contents

Building Data Streaming Applications with Apache Kafka

By : Singh, Kumar

3.8 (4)

Building Data Streaming Applications with Apache Kafka

3.8 (4)

By: Singh, Kumar

Overview of this book

Apache Kafka is a popular distributed streaming platform that acts as a messaging queue or an enterprise messaging system. It lets you publish and subscribe to a stream of records, and process them in a fault-tolerant way as they occur. This book is a comprehensive guide to designing and architecting enterprise-grade streaming applications using Apache Kafka and other big data tools. It includes best practices for building such applications, and tackles some common challenges such as how to use Kafka efficiently and handle high data volumes with ease. This book first takes you through understanding the type messaging system and then provides a thorough introduction to Apache Kafka and its internal details. The second part of the book takes you through designing streaming application using various frameworks and tools such as Apache Spark, Apache Storm, and more. Once you grasp the basics, we will take you through more advanced concepts in Apache Kafka such as capacity planning and security. By the end of this book, you will have all the information you need to be comfortable with using Apache Kafka, and to design efficient streaming data applications with it.

Preface

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Free Chapter

Introduction to Messaging Systems

Introduction to Messaging Systems

Understanding the principles of messaging systems

Understanding messaging systems

Peeking into a point-to-point messaging system

Publish-subscribe messaging system

Advance Queuing Messaging Protocol

Using messaging systems in big data streaming applications

Summary

Introducing Kafka the Distributed Messaging Platform

Introducing Kafka the Distributed Messaging Platform

Kafka origins

Kafka's architecture

Message topics

Message partitions

Replication and replicated logs

Message producers

Message consumers

Role of Zookeeper

Summary

Deep Dive into Kafka Producers

Deep Dive into Kafka Producers

Kafka producer internals

Kafka Producer APIs

Java Kafka producer example

Common messaging publishing patterns

Best practices

Summary

Deep Dive into Kafka Consumers

Deep Dive into Kafka Consumers

Kafka consumer internals

Kafka consumer APIs

Java Kafka consumer

Scala Kafka consumer

Common message consuming patterns

Best practices

Summary

Building Spark Streaming Applications with Kafka

Building Spark Streaming Applications with Kafka

Introduction to Spark

Spark Streaming

Use case log processing - fraud IP detection

Producer

Summary

Building Storm Applications with Kafka

Building Storm Applications with Kafka

Introduction to Apache Storm

Introduction to Apache Heron

Integrating Apache Kafka with Apache Storm - Java

Integrating Apache Kafka with Apache Storm - Scala

Use case – log processing in Storm, Kafka, Hive

Summary

Using Kafka with Confluent Platform

Using Kafka with Confluent Platform

Introduction to Confluent Platform

Deep driving into Confluent architecture

Understanding Kafka Connect and Kafka Stream

Playing with Avro using Schema Registry

Moving Kafka data to HDFS

Summary

Building ETL Pipelines Using Kafka

Building ETL Pipelines Using Kafka

Considerations for using Kafka in ETL pipelines

Introducing Kafka Connect

Deep dive into Kafka Connect

Introductory examples of using Kafka Connect

Kafka Connect common use cases

Summary

Building Streaming Applications Using Kafka Streams

Building Streaming Applications Using Kafka Streams

Introduction to Kafka Streams

Kafka Stream architecture

Integrated framework advantages

Understanding tables and Streams together

Use case example of Kafka Streams

Summary

Kafka Cluster Deployment

Kafka Cluster Deployment

Kafka cluster internals

Capacity planning

Single cluster deployment

Multicluster deployment

Decommissioning brokers

Data migration

Summary

Using Kafka in Big Data Applications

Using Kafka in Big Data Applications

Managing high volumes in Kafka

Kafka message delivery semantics

Big data and Kafka common usage patterns

Kafka and data governance

Alerting and monitoring

Useful Kafka matrices

Summary

Securing Kafka

Securing Kafka

An overview of securing Kafka

Wire encryption using SSL

Kerberos SASL for authentication

Understanding ACL and authorization

Understanding Zookeeper authentication

Apache Ranger for authorization

Best practices

Summary

Streaming Application Design Considerations

Streaming Application Design Considerations

Latency and throughput

Data and state persistence

Data sources

External data lookups

Data formats

Data serialization

Level of parallelism

Out-of-order events

Message processing semantics

Summary

Deep Dive into Kafka Producers

In previous chapters, you have learned about messaging systems and Kafka architecture. While it is a good start, we will now take a deeper look into Kafka producers. Kafka can be used as a message queue, message bus, or data storage system. Irrespective of how Kafka is used in your enterprise, you will need an application system that can write data to the Kafka cluster. Such a system is called a producer. As the name suggests, they are the source or producers of messages for Kafka topics. Kafka producers publish messages as per Kafka protocols defined by the makers of Kafka. This chapter is all about producers, their internal working, examples of writing producers using Java or Scala APIs, and some of the best practices of writing Kafka APIs. We will cover the following topics in this chapter:

Internals of a Kafka producer
The Kafka Producer API...

CONTINUE READING

83

Tech Concepts

36

Programming languages

73

Tech Tools

Unlimited access to the largest independent learning library in tech of over 8,000 expert-authored tech books and videos.

Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.

50+ new titles added per month and exclusive early access to books as they are being written.

Building Data Streaming Applications with Apache Kafka

Search

Your notes and bookmarks