Book Image

Building Data Streaming Applications with Apache Kafka

By : Chanchal Singh, Manish Kumar
Book Image

Building Data Streaming Applications with Apache Kafka

By: Chanchal Singh, Manish Kumar

Overview of this book

Apache Kafka is a popular distributed streaming platform that acts as a messaging queue or an enterprise messaging system. It lets you publish and subscribe to a stream of records, and process them in a fault-tolerant way as they occur. This book is a comprehensive guide to designing and architecting enterprise-grade streaming applications using Apache Kafka and other big data tools. It includes best practices for building such applications, and tackles some common challenges such as how to use Kafka efficiently and handle high data volumes with ease. This book first takes you through understanding the type messaging system and then provides a thorough introduction to Apache Kafka and its internal details. The second part of the book takes you through designing streaming application using various frameworks and tools such as Apache Spark, Apache Storm, and more. Once you grasp the basics, we will take you through more advanced concepts in Apache Kafka such as capacity planning and security. By the end of this book, you will have all the information you need to be comfortable with using Apache Kafka, and to design efficient streaming data applications with it.

Related Content you might be interested in

Current Title:

Small book image
Building Data Streaming Applications with Apache Kafka
Chanchal Singh | Manish Kumar
Aug, 2017 | 278 pages
No titles found
Table of Contents (14 chapters)
Preface
Preface
What this book covers
What you need for this book
Who this book is for
Conventions
Reader feedback
Customer support
Free Chapter
1
Introduction to Messaging Systems
Introduction to Messaging Systems
Understanding the principles of messaging systems
Understanding messaging systems
Peeking into a point-to-point messaging system
Publish-subscribe messaging system
Advance Queuing Messaging Protocol
Using messaging systems in big data streaming applications
Summary
2
Introducing Kafka the Distributed Messaging Platform
Introducing Kafka the Distributed Messaging Platform
Kafka origins
Kafka's architecture
Message topics
Message partitions
Replication and replicated logs
Message producers
Message consumers
Role of Zookeeper
Summary
3
Deep Dive into Kafka Producers
Deep Dive into Kafka Producers
Kafka producer internals
Kafka Producer APIs
Java Kafka producer example
Common messaging publishing patterns
Best practices
Summary
4
Deep Dive into Kafka Consumers
Deep Dive into Kafka Consumers
Kafka consumer internals
Kafka consumer APIs
Java Kafka consumer
Scala Kafka consumer
Common message consuming patterns
Best practices
Summary
5
Building Spark Streaming Applications with Kafka
Building Spark Streaming Applications with Kafka
Introduction to Spark
Spark Streaming
Use case log processing - fraud IP detection
Producer
Summary
6
Building Storm Applications with Kafka
Building Storm Applications with Kafka
Introduction to Apache Storm
Introduction to Apache Heron
Integrating Apache Kafka with Apache Storm - Java
Integrating Apache Kafka with Apache Storm - Scala
Use case – log processing in Storm, Kafka, Hive
Summary
7
Using Kafka with Confluent Platform
Using Kafka with Confluent Platform
Introduction to Confluent Platform
Deep driving into Confluent architecture
Understanding Kafka Connect and Kafka Stream
Playing with Avro using Schema Registry
Moving Kafka data to HDFS
Summary
8
Building ETL Pipelines Using Kafka
Building ETL Pipelines Using Kafka
Considerations for using Kafka in ETL pipelines
Introducing Kafka Connect
Deep dive into Kafka Connect
Introductory examples of using Kafka Connect
Kafka Connect common use cases
Summary
9
Building Streaming Applications Using Kafka Streams
Building Streaming Applications Using Kafka Streams
Introduction to Kafka Streams
Kafka Stream architecture
Integrated framework advantages
Understanding tables and Streams together
Use case example of Kafka Streams
Summary
10
Kafka Cluster Deployment
Kafka Cluster Deployment
Kafka cluster internals
Capacity planning
Single cluster deployment
Multicluster deployment
Decommissioning brokers
Data migration
Summary
11
Using Kafka in Big Data Applications
Using Kafka in Big Data Applications
Managing high volumes in Kafka
Kafka message delivery semantics
Big data and Kafka common usage patterns
Kafka and data governance
Alerting and monitoring
Useful Kafka matrices
Summary
12
Securing Kafka
Securing Kafka
An overview of securing Kafka
Wire encryption using SSL
Kerberos SASL for authentication
Understanding ACL and authorization
Understanding Zookeeper authentication
Apache Ranger for authorization
Best practices
Summary
13
Streaming Application Design Considerations
Streaming Application Design Considerations
Latency and throughput
Data and state persistence
Data sources
External data lookups
Data formats
Data serialization
Level of parallelism
Out-of-order events
Message processing semantics
Summary

Big data and Kafka common usage patterns

In the big data world, Kafka can be used in multiple ways. One of the common usage patterns of Kafka is to use it as a streaming data platform. It supports storing streaming data from varied sources, and that data can later be processed in real time or in batch.

The following diagram shows a typical pattern for using Kafka as a streaming data platform:

Kafka as streaming data platform

The previous diagram depicts how Kafka can be used for storing events from a variety of data sources. Of course, the data ingestion mechanism would differ depending upon the type of data sources. However, once data is stored in Kafka topics, it can be used in data search engines, real-time processing, or alerting and even for batch processing.

Batch processing engines, such as Gobblin, read data from Kafka and use Hadoop MapReduce to store data in Hadoop...
Previous Section
End of Section 4
Next Section