Apache Kafka A-Z with Hands-On Learning [Video]

Book Image

Apache Kafka A-Z with Hands-On Learning [Video]

By : Learnkart Technology Private Limited

Book Image

Apache Kafka A-Z with Hands-On Learning [Video]

By: Learnkart Technology Private Limited

Overview of this book

Do you want to learn Apache Kafka to store and process multiple, nonstop streams of information faster? Or do you want to prepare for the Apache Kafka certification exam? This course is an ideal resource to learn Apache Kafka and to reach your goals. The course starts with an introduction to big data and Apache Kafka. Next, you will get into the details of a Kafka producer and consumer and grasp the techniques to tune the performance of Kafka operations. Moving along, you will explore the Kafka architecture and learn how to administer Kafka effectively. Next, you will learn to monitor server, infrastructure, and application and understand how the Schema Registry works in Kafka. Later, you will get to grips with Kafka Streams and Connectors and learn the process of integrating Kafka with Apache Storm, Spark, and Flume. Towards the end, you will learn how to perform various admin tasks using the admin clients and uncover useful tips to secure Kafka. By the end of this course, you will have developed the Kafka skills needed to handle real-time data feed and to pass the Apache Kafka certification exam.

Free Chapter

Course Introduction

Getting Started with Big data and Apache Kafka

Learning Objectives

Data Evolution over Time

Big Data Overview

Big Data Analytics

Messaging System

Introduction and Features of Kafka

Kafka Use Cases

Kafka Terminologies

Kafka Components

Kafka Architecture

Kafka Industry Use Cases

Demo - Installing Zookeeper

Demo - Installing Kafka

Demo – Setting up a Single Node and Single Broker Cluster

Demo – Setting up a Multi-node Single Broker Cluster

Kafka Producers

Learning Objective

Overview of a Kafka Producer and Its Architecture

Kafka Producer Configuration

Kafka Producer Optional Configuration

Kafka Producer Configuration Objects

Demo - Creating a Kafka Producer

Sending Messages

Serializers and Custom Serializers

Demo - Creating a Custom Serializer

Serializers Challenges and Serializing Using Apache Avro

Demo - Serializing Using Apache Avro

Demo – Setting up a Custom Partition

Kafka Consumers

Learning Objective

Overview of Kafka Consumers

Consumer Groups

Partition Rebalance and Creating a Kafka Consumer

Poll Loop and Its Functioning

Configuring a Kafka Consumer - Part 1

Configuring a Kafka Consumer - Part 2

Demo - Creating a Kafka Consumer

Commit and Offset

Ways of Committing Offset - Automatic Offset

Ways of Committing Offset - Commit Current Offset

Ways of Committing Offset - Asynchronous Commit

Ways of Committing Offset - Combining Synchronous and Asynchronous Commits

Ways of Committing Offset - Commit Specified Offset

Rebalance Listeners

Consuming Records with Specific Offset

Demo - Creating and Using Custom Deserializer

Kafka Operations and Performance Tuning

Learning Objective

Kafka Internals Overview

Replication and Replica Types

Preferred Ladder, Request, and Request Processing

Types of Requests

Partition Allocation, File Management, and Segments

File Format, Index, and Compaction

Kafka Reliability and Reliability Methods

Broker Configuration for Replication

Producer in a Reliable System

Consumer in a Reliable System

Kafka Cluster Architectures and Administering Kafka

Learning Objective

Cluster Mirroring

Multi-cluster Architecture - Hub, Spokes, and Active Architecture – Part 1

Multi-cluster Architecture – Active/Stand-by Architecture and Stretch Clusters Part – 2

Configuring MirrorMaker – Part 1

Configuring MirrorMaker – Part 2

Demo - Setting up MirrorMaker

Administering Kafka - Topic Operations

Administering Kafka - Consumer Group Operations

Dynamic Configuration Changes

Partition Management

Console Producer Tool

Console Consumer Tool

Monitoring Kafka and Schema Registry

Learning Objective

Monitoring and Its Importance

Server or Infrastructure Monitoring and Application Monitoring

Monitoring Kafka

Kafka Broker Metrics – Under-replicated Partitions – Part 1

Kafka Broker Metrics – Others– Part 2

Topic and Partition Specific Metrics

Logging and Client Monitoring

Producer and Consumer Metrics

Quotas and Lag Monitoring

Monitoring Dashboard

Demo - Setting up an Open Source Health Monitor

Kafka Schema Registry

Kafka Components and Its Architecture

Kafka Schema Registry - Internal working and Use cases

Working of the Kafka Schema Registry

Demo – Using the Kafka Schema Registry with Kafka

Kafka Streams and Kafka Connectors

Learning Objective

Kafka Streams Overview

Kafka Streams Architecture and Working

Kafka Streams Components

Kafka Streams Architecture Tasks, Threading Model and Local State Store

Kafka Streams Architecture - Record Buffer

Memory Management and Streaming Data Pipeline

Kafka Streams Domain Specific Language (DSL)

KStream Operations

KTable Operations

KTable Operations

Aggregation and Windowing

Processor Topology and Stream Processor

Stream and Processor Application Programming Interfaces (APIs)

Processor Application Programming Interfaces (APIs) and Creating Topology

Kafka Connectors

Stand-alone and Sink Connector Configuration

Running Kafka Connect

Kafka Connector Distributed Mode

Hypertext Transfer Protocol (HTTP) Representational State Transfer (REST) Interface

Demo - Kafka Connector

Demo - Creating an Application Using Kafka Streams

Integration of Kafka with Storm

Learning Objective

Real-time Analytics

Apache Storm Architecture

Apache Storm Components

Apache Storm Topology Part - 1

Apache Storm Topology Part - 2

Integration of Apache Storm and Kafka

Demo - Simple Stand-alone Application Using Kafka and Storm

Kafka Integration with Spark and Flume

Learning Objective

Introduction to Spark

Spark Components

Resilient Distributed Dataset

Resilient Distributed Dataset (RDD) Operations - Transformation - Map, FlatMap, and Filter

RDD Operations - Transformation - Join, Distinct, First, and Take

Resilient Distributed Dataset (RDD) Operations - Actions

Data Sets and SparkSession

Data Sets and Spark Session Operations

Integrating Spark with Kafka

Demo - Running a Small Stand-alone Application in Spark with Kafka

Flume Connectors

Configuring Flume Kafka to Hadoop Distributed File System (HDFS)

Demo - Creating a Flume Agent and Sending Data from Kafka to Hadoop Distributed File System (HDFS)

Admin Client and Securing Kafka

Learning Objective

Demo - Performing Various Admin Tasks Using AdminClient

Kafka Security Components

Simple Authentication and Security Layer (SASL)

Configuring Simple Authentication and Security Layer (SASL) in Kafka

Securing Using Access Control Lists (ACLs)

Customer Reviews

5 star

0

4 star

0

3 star

0

2 star

0

1 star

0

Chapter 1

Course Introduction

Section 1

Introduction

This video presents an overview of the course.