Book Image

Mastering Hadoop 3

By : Chanchal Singh, Manish Kumar
Book Image

Mastering Hadoop 3

By: Chanchal Singh, Manish Kumar

Overview of this book

Apache Hadoop is one of the most popular big data solutions for distributed storage and for processing large chunks of data. With Hadoop 3, Apache promises to provide a high-performance, more fault-tolerant, and highly efficient big data processing platform, with a focus on improved scalability and increased efficiency. With this guide, you’ll understand advanced concepts of the Hadoop ecosystem tool. You’ll learn how Hadoop works internally, study advanced concepts of different ecosystem tools, discover solutions to real-world use cases, and understand how to secure your cluster. It will then walk you through HDFS, YARN, MapReduce, and Hadoop 3 concepts. You’ll be able to address common challenges like using Kafka efficiently, designing low latency, reliable message delivery Kafka systems, and handling high data volumes. As you advance, you’ll discover how to address major challenges when building an enterprise-grade messaging system, and how to use different stream processing systems along with Kafka to fulfil your enterprise goals. By the end of this book, you’ll have a complete understanding of how components in the Hadoop ecosystem are effectively integrated to implement a fast and reliable data pipeline, and you’ll be equipped to tackle a range of real-world problems in data pipelines.
Table of Contents (23 chapters)
Title Page
Dedication
About Packt
Foreword
Contributors
Preface
Index

List of security features that have been worked upon in Hadoop 3.0


The following is the a of the JIRAs that have been solved or worked upon with the Hadoop 3.0 release:

AliyunOSS: update oss-sdk version to 3.0.0 MajorResolvedFixed:

Issue Type

Issue key

Issue ID

Parent ID

Summary

Bug

HADOOP-15866

13192984

 

Renamed HADOOP_SECURITY_GROUP_SHELL_COMMAND_TIMEOUT keys break compatibility

Task

HADOOP-15816

13189089

 

Upgrade Apache ZooKeeper version due to security concerns

Bug

HADOOP-15861

13192122

 

Move DelegationTokenIssuer to the right path

Bug

HADOOP-15523

13164833

 

Shell command timeout given is in seconds whereas it is taken as millisec while scheduling

Improvement

HADOOP-15609

13172372

 

Retry KMS calls when SSLHandshakeException occurs

Improvement

HADOOP-15804

13188359

 

Upgrade to commons-compress 1.18

Bug

HADOOP-15698

13181280

 

KMS log4j is not initialized properly at startup

Bug

HADOOP-15864

13192767

 

Job submitter/executor fail when SBN domain name can not resolved

Subtask

HADOOP-15607

13172317

12989378

AliyunOSS: fix duplicated...