Book Image

Mastering Hadoop 3

By : Chanchal Singh, Manish Kumar
Book Image

Mastering Hadoop 3

By: Chanchal Singh, Manish Kumar

Overview of this book

Apache Hadoop is one of the most popular big data solutions for distributed storage and for processing large chunks of data. With Hadoop 3, Apache promises to provide a high-performance, more fault-tolerant, and highly efficient big data processing platform, with a focus on improved scalability and increased efficiency. With this guide, you’ll understand advanced concepts of the Hadoop ecosystem tool. You’ll learn how Hadoop works internally, study advanced concepts of different ecosystem tools, discover solutions to real-world use cases, and understand how to secure your cluster. It will then walk you through HDFS, YARN, MapReduce, and Hadoop 3 concepts. You’ll be able to address common challenges like using Kafka efficiently, designing low latency, reliable message delivery Kafka systems, and handling high data volumes. As you advance, you’ll discover how to address major challenges when building an enterprise-grade messaging system, and how to use different stream processing systems along with Kafka to fulfil your enterprise goals. By the end of this book, you’ll have a complete understanding of how components in the Hadoop ecosystem are effectively integrated to implement a fast and reliable data pipeline, and you’ll be equipped to tackle a range of real-world problems in data pipelines.
Table of Contents (23 chapters)
Title Page
Dedication
About Packt
Foreword
Contributors
Preface
Index

HDFS common interfaces


In this section, we will dig into the HDFS Java interfaces and APIs that can be used to interact with the HDFS File System. We will also to focus on the method implementation of FileSystem classes so that you can understand how to write your own implementation if needed. This will also help you when you test your program against certain test cases. 

HDFS read 

We will talk about two approaches to reading a file and will discuss when to use what. In the first approach, we can use the URL class, which is part of the java.net package and can be used to read files stored on HDFS. The URL calls setURLStreamHandlerFactory(), which requires an instance of FsUrlStreamHandlerFactory(). This initialization is a part of a static block that is executed before any instance creation. This method is in a static block because it can only be called once per JVM and hence, if any third-party program sets URLStreamHandlerFactory, we won't be able to use it for reading files from HDFS:

static...