Book Image

Apache Kafka - Real-time Stream Processing (Master Class) [Video]

By : Prashant Kumar Pandey
3.5 (2)
Book Image

Apache Kafka - Real-time Stream Processing (Master Class) [Video]

3.5 (2)
By: Prashant Kumar Pandey

Overview of this book

If you want to understand the concept of stream processing, this course is for you. Using Kafka, the course will help you get to grips with real-time stream processing and enable you to apply that knowledge to learn Kafka programming techniques. This course uses the Kafka Streams library available in Apache Kafka 2.x. All the source code and examples on Apache Kafka 2.3 open-source distribution have been tested. You'll understand and explore Confluent Platform functionalities such as Schema Registry and Avro Serdes using the Confluent Community Version. This course makes extensive use of IntelliJ IDEA and Apache Maven as the preferred development IDE. You'll leverage Log4J2 and JUnit5 for industry-standard log implementation in your application and implementing unit test cases, respectively. The code bundle is available at https://github.com/PacktPublishing/Apache-Kafka---Real-time-Stream-Processing-Master-Class

Related Content you might be interested in

Current Title:

Small book image
Apache Kafka - Real-time Stream Processing (Master Class) [Video]
Prashant Kumar Pandey
Apr, 2020 | 11 hours 0 minutes
No titles found
Table of Contents (14 chapters)
Free Chapter
1
Before you start
Introduction
About the Course
What do you Need for this Course?
Debugging Problems
2
Introduction to Real-time Streams
Emergence of Bigdata - A Quick Recap
Conception of Event Streams
Real-time Streaming - Use Cases
Real-time Streaming Challenges
Real-time Streaming Design Consideration
Section Summary
3
Enter the world of Apache Kafka
What is Apache Kafka?
Kafka Storage Architecture
Kafka Cluster Architecture
Kafka Work Distribution Architecture - Part 1
Kafka Work Distribution Architecture - Part 2
Section Summary
4
Creating Real-time Streams
Streaming into Kafka
Kafka Producers - Quick Start
Kafka Producer Internals
Scaling Kafka Producer
Advanced Kafka Producers (Exactly Once)
Advanced Kafka Producer (Implementing Transaction)
Kafka Producer - Micro Project
Kafka Producer - Final Note and References
5
Enter the Stream Processing
Stream Processing in Apache Kafka
Kafka Consumer - Practical Introduction
Kafka Consumer - Scalability, Fault tolerance and Missing Features
Kafka Streams API - Quick Start
Creating Streams Topology
Implementing Streams Topology
Kafka Streams Architecture
Section Summary and References
6
Foundation for Real Life Implementations
Introduction to Types and Serialization in Kafka
JSON Schema to POJO for JSON Serdes
Creating and Using JSON Serdes
AVRO Schema to POJO for AVRO Serdes
Creating and using AVRO schema in Producers
Creating and using AVRO schema in Kafka Streams
Section Summary and References
7
States and Stores
Understanding States and State Stores
Creating your First State Store
Caution with States
State Store Fault Tolerance
Section Summary and References
8
KTable - An Update Stream
Introducing KTable
Creating your First Update Stream - Ktable
Table Caching and Emit Rates
Introducing GlobalKTable
9
Real-time Aggregates
Computing Your First Aggregate - Real-time Streaming Word Count
Streaming Aggregates - Core Concept
KStream Aggregation using Reduce()
KStream Aggregation using Aggregate()
Common Mistakes in Aggregation
Count on KTable
KTable Aggregation using Aggregate()
10
Timestamps and Windows
Timestamps and Timestamp Extractors
Creating Tumbling Windows
Stream Time and Grace Period
Supressing Intermediate Results
Creating Hopping Windows
Creating Session Windows
11
Joining Streams and Tables
Streaming Joins
Joining a KStrem to another KStream
Joining a KTable to another KTable
Joining a KStream to a KTable and GlobalKTable
Mixing Joins with Aggregates - Computing Top 3
Mixing Joins with Aggregates - Advert CTR
12
Testing Streams Application
How to test a Stream Processing Application
Unit Testing Your Topology
13
Interactive Query and Micro-Service Responses
Introducing Micro-services Requirement
Understanding Local Vs Remote State Store
Implementing Interactive Query Micro-service
14
Appendix
Setting up Apache Kafka Development Environment
Chapter 4
Creating Real-time Streams
Content Locked
Section 3
Kafka Producer Internals
In this lecture, you will learn about the internals of Kafka Producer API. While the API methods are straightforward, but a lot of things happen behind the scenes. This lecture will help you understand what happens under the hood and build solid conceptual foundation.
Previous Section
End of Section 3
Next Section