Book Image

Building Data Streaming Applications with Apache Kafka

By : Chanchal Singh, Manish Kumar
Book Image

Building Data Streaming Applications with Apache Kafka

By: Chanchal Singh, Manish Kumar

Overview of this book

Apache Kafka is a popular distributed streaming platform that acts as a messaging queue or an enterprise messaging system. It lets you publish and subscribe to a stream of records, and process them in a fault-tolerant way as they occur. This book is a comprehensive guide to designing and architecting enterprise-grade streaming applications using Apache Kafka and other big data tools. It includes best practices for building such applications, and tackles some common challenges such as how to use Kafka efficiently and handle high data volumes with ease. This book first takes you through understanding the type messaging system and then provides a thorough introduction to Apache Kafka and its internal details. The second part of the book takes you through designing streaming application using various frameworks and tools such as Apache Spark, Apache Storm, and more. Once you grasp the basics, we will take you through more advanced concepts in Apache Kafka such as capacity planning and security. By the end of this book, you will have all the information you need to be comfortable with using Apache Kafka, and to design efficient streaming data applications with it.

Related Content you might be interested in

Current Title:

Small book image
Building Data Streaming Applications with Apache Kafka
Chanchal Singh | Manish Kumar
Aug, 2017 | 278 pages
Small book image
Mastering Hadoop 3
Manish Kumar | Chanchal Singh
Feb, 2019 | 544 pages
Small book image
Mastering Apache Storm
Ankit Jain
Aug, 2017 | 284 pages
Small book image
Apache Kafka 1.0 Cookbook
Raúl Estrada
Dec, 2017 | 250 pages
Table of Contents (14 chapters)
Preface
Preface
What this book covers
What you need for this book
Who this book is for
Conventions
Reader feedback
Customer support
Free Chapter
1
Introduction to Messaging Systems
Introduction to Messaging Systems
Understanding the principles of messaging systems
Understanding messaging systems
Peeking into a point-to-point messaging system
Publish-subscribe messaging system
Advance Queuing Messaging Protocol
Using messaging systems in big data streaming applications
Summary
2
Introducing Kafka the Distributed Messaging Platform
Introducing Kafka the Distributed Messaging Platform
Kafka origins
Kafka's architecture
Message topics
Message partitions
Replication and replicated logs
Message producers
Message consumers
Role of Zookeeper
Summary
3
Deep Dive into Kafka Producers
Deep Dive into Kafka Producers
Kafka producer internals
Kafka Producer APIs
Java Kafka producer example
Common messaging publishing patterns
Best practices
Summary
4
Deep Dive into Kafka Consumers
Deep Dive into Kafka Consumers
Kafka consumer internals
Kafka consumer APIs
Java Kafka consumer
Scala Kafka consumer
Common message consuming patterns
Best practices
Summary
5
Building Spark Streaming Applications with Kafka
Building Spark Streaming Applications with Kafka
Introduction to Spark
Spark Streaming
Use case log processing - fraud IP detection
Producer
Summary
6
Building Storm Applications with Kafka
Building Storm Applications with Kafka
Introduction to Apache Storm
Introduction to Apache Heron
Integrating Apache Kafka with Apache Storm - Java
Integrating Apache Kafka with Apache Storm - Scala
Use case – log processing in Storm, Kafka, Hive
Summary
7
Using Kafka with Confluent Platform
Using Kafka with Confluent Platform
Introduction to Confluent Platform
Deep driving into Confluent architecture
Understanding Kafka Connect and Kafka Stream
Playing with Avro using Schema Registry
Moving Kafka data to HDFS
Summary
8
Building ETL Pipelines Using Kafka
Building ETL Pipelines Using Kafka
Considerations for using Kafka in ETL pipelines
Introducing Kafka Connect
Deep dive into Kafka Connect
Introductory examples of using Kafka Connect
Kafka Connect common use cases
Summary
9
Building Streaming Applications Using Kafka Streams
Building Streaming Applications Using Kafka Streams
Introduction to Kafka Streams
Kafka Stream architecture
Integrated framework advantages
Understanding tables and Streams together
Use case example of Kafka Streams
Summary
10
Kafka Cluster Deployment
Kafka Cluster Deployment
Kafka cluster internals
Capacity planning
Single cluster deployment
Multicluster deployment
Decommissioning brokers
Data migration
Summary
11
Using Kafka in Big Data Applications
Using Kafka in Big Data Applications
Managing high volumes in Kafka
Kafka message delivery semantics
Big data and Kafka common usage patterns
Kafka and data governance
Alerting and monitoring
Useful Kafka matrices
Summary
12
Securing Kafka
Securing Kafka
An overview of securing Kafka
Wire encryption using SSL
Kerberos SASL for authentication
Understanding ACL and authorization
Understanding Zookeeper authentication
Apache Ranger for authorization
Best practices
Summary
13
Streaming Application Design Considerations
Streaming Application Design Considerations
Latency and throughput
Data and state persistence
Data sources
External data lookups
Data formats
Data serialization
Level of parallelism
Out-of-order events
Message processing semantics
Summary

Producer 

You can use IntelliJ or Eclipse to build a producer application. This producer reads a log file taken from an Apache project which contains detailed records like:

64.242.88.10 - - [08/Mar/2004:07:54:30 -0800] "GET /twiki/bin/edit/Main/Unknown_local_recipient_reject_code?topicparent=Main.ConfigurationVariables HTTP/1.1" 401 12846

You can have just one record in the test file and the producer will produce records by generating random IPs and replace it with existing. So, we will have millions of distinct records with unique IP addresses.

Record columns are separated by space delimiters, which we change to commas in producer. The first column represents the IP address or the domain name which will be used to detect whether the request was from a fraud client. The following is the Java Kafka producer which remembers logs.

...
Previous Section
End of Section 5
Next Section