Book Image

Mastering Hadoop 3

By : Chanchal Singh, Manish Kumar
Book Image

Mastering Hadoop 3

By: Chanchal Singh, Manish Kumar

Overview of this book

Apache Hadoop is one of the most popular big data solutions for distributed storage and for processing large chunks of data. With Hadoop 3, Apache promises to provide a high-performance, more fault-tolerant, and highly efficient big data processing platform, with a focus on improved scalability and increased efficiency. With this guide, you’ll understand advanced concepts of the Hadoop ecosystem tool. You’ll learn how Hadoop works internally, study advanced concepts of different ecosystem tools, discover solutions to real-world use cases, and understand how to secure your cluster. It will then walk you through HDFS, YARN, MapReduce, and Hadoop 3 concepts. You’ll be able to address common challenges like using Kafka efficiently, designing low latency, reliable message delivery Kafka systems, and handling high data volumes. As you advance, you’ll discover how to address major challenges when building an enterprise-grade messaging system, and how to use different stream processing systems along with Kafka to fulfil your enterprise goals. By the end of this book, you’ll have a complete understanding of how components in the Hadoop ecosystem are effectively integrated to implement a fast and reliable data pipeline, and you’ll be equipped to tackle a range of real-world problems in data pipelines.
Table of Contents (23 chapters)
Title Page
Dedication
About Packt
Foreword
Contributors
Preface
Index

Spark


Hadoop has been used as a processing framework for large datasets for the past decade and it has brought tremendous value and cost saving to organizations. MapReduce has evolved over a time but it is not efficient for a few use cases like near real-time computation, multi-pass computation, which is iterative processing, and so on. Every time the data is processed, it has to be written into the disk and then you have to pick data from disk for further processing. Along with this, if we need to add additional use cases which require libraries such as Mahout and Apache Storm, then it has to be integrated separately in the Hadoop cluster. Spark is a distributed data processing framework that provides functional APIs for manipulating data at scale, in-memory data caching, and reusability of datasets. Spark utilizes the concept of the direct acyclic graph (DAG), which is a data lineage graph that helps in recomputing tasks in case of failure. Spark supports a number of file formats and rich...