Book Image

Mastering Hadoop 3

By : Chanchal Singh, Manish Kumar
Book Image

Mastering Hadoop 3

By: Chanchal Singh, Manish Kumar

Overview of this book

Apache Hadoop is one of the most popular big data solutions for distributed storage and for processing large chunks of data. With Hadoop 3, Apache promises to provide a high-performance, more fault-tolerant, and highly efficient big data processing platform, with a focus on improved scalability and increased efficiency. With this guide, you’ll understand advanced concepts of the Hadoop ecosystem tool. You’ll learn how Hadoop works internally, study advanced concepts of different ecosystem tools, discover solutions to real-world use cases, and understand how to secure your cluster. It will then walk you through HDFS, YARN, MapReduce, and Hadoop 3 concepts. You’ll be able to address common challenges like using Kafka efficiently, designing low latency, reliable message delivery Kafka systems, and handling high data volumes. As you advance, you’ll discover how to address major challenges when building an enterprise-grade messaging system, and how to use different stream processing systems along with Kafka to fulfil your enterprise goals. By the end of this book, you’ll have a complete understanding of how components in the Hadoop ecosystem are effectively integrated to implement a fast and reliable data pipeline, and you’ll be equipped to tackle a range of real-world problems in data pipelines.
Table of Contents (23 chapters)
Title Page
Dedication
About Packt
Foreword
Contributors
Preface
Index

Managing disk-skewed data in Hadoop 3.x


Over any period of time, when you're producing a Hadoop cluster, there is always a need to manage disks on DataNodes. It could be the case that you must replace corrupted disks or you must add more disks for more data volumes. Another possibility is that your disks volumes vary in same data nodes. All such cases would result in uneven data distribution across all of the disks in a DataNode. Another reason that can result in uneven data distribution is round robin-based disk writes and random deletes.

 

 

To prevent such problems from occurring prior to the release of Hadoop 3, Hadoop administrators were applying methods that were far from ideal. One solution was to shut down your data node and use the UNIX mv command to move block replicas along with supported metadata files from one directory to another directory. Each of those directories should be using different disks. You need to ensure that subdirectory names are not changed, otherwise, upon rebooting...