Browse Library
Sign In Start Free Trial
Close icon Search icon
Account
Browse Library Sign In Start Free Trial

Add to playlist

Create a Playlist

Modal Close icon
You need to login to use this feature.
  • Book Overview & Buying Mastering Hadoop
  • Table Of Contents Toc
Mastering Hadoop

Mastering Hadoop

By : Karanth
4 (3)
Buy this Book
close
close
Mastering Hadoop

Mastering Hadoop

4 (3)
By: Karanth
Buy this Book

Overview of this book

Do you want to broaden your Hadoop skill set and take your knowledge to the next level? Do you wish to enhance your knowledge of Hadoop to solve challenging data processing problems? Are your Hadoop jobs, Pig scripts, or Hive queries not working as fast as you intend? Are you looking to understand the benefits of upgrading Hadoop? If the answer is yes to any of these, this book is for you. It assumes novice-level familiarity with Hadoop.
Table of Contents (15 chapters)
close
close
close
Preface
Icon
Preface
Icon
What this book covers
Icon
What you need for this book?
Icon
Who this book is for
Icon
Conventions
Icon
Reader feedback
Icon
Customer support
Lock Free Chapter
1
1. Hadoop 2.X
Icon
1. Hadoop 2.X
Icon
The inception of Hadoop
Icon
The evolution of Hadoop
Icon
Hadoop 2.X
Icon
Hadoop distributions
Icon
Summary
2
2. Advanced MapReduce
Icon
2. Advanced MapReduce
Icon
MapReduce input
Icon
The RecordReader class
Icon
Hadoop's "small files" problem
Icon
Filtering inputs
Icon
The Map task
Icon
The Reduce task
Icon
MapReduce output
Icon
MapReduce job counters
Icon
Handling data joins
Icon
Summary
3
3. Advanced Pig
Icon
3. Advanced Pig
Icon
Pig versus SQL
Icon
Different modes of execution
Icon
Complex data types in Pig
Icon
Compiling Pig scripts
Icon
Development and debugging aids
Icon
The advanced Pig operators
Icon
User-defined functions
Icon
Pig performance optimizations
Icon
Best practices
Icon
Summary
4
4. Advanced Hive
Icon
4. Advanced Hive
Icon
The Hive architecture
Icon
Data types
Icon
File formats
Icon
The data model
Icon
Hive query optimizers
Icon
Advanced DML
Icon
UDF, UDAF, and UDTF
Icon
Summary
5
5. Serialization and Hadoop I/O
Icon
5. Serialization and Hadoop I/O
Icon
Data serialization in Hadoop
Icon
Avro serialization
Icon
File formats
Icon
Compression
Icon
Summary
6
6. YARN – Bringing Other Paradigms to Hadoop
Icon
6. YARN – Bringing Other Paradigms to Hadoop
Icon
The YARN architecture
Icon
Developing YARN applications
Icon
Monitoring YARN
Icon
Job scheduling in YARN
Icon
YARN commands
Icon
Summary
7
7. Storm on YARN – Low Latency Processing in Hadoop
Icon
7. Storm on YARN – Low Latency Processing in Hadoop
Icon
Batch processing versus streaming
Icon
Apache Storm
Icon
Storm on YARN
Icon
Summary
8
8. Hadoop on the Cloud
Icon
8. Hadoop on the Cloud
Icon
Cloud computing characteristics
Icon
Hadoop on the cloud
Icon
Amazon Elastic MapReduce (EMR)
Icon
Summary
9
9. HDFS Replacements
chevron up
Icon
9. HDFS Replacements
Icon
HDFS – advantages and drawbacks
Icon
Amazon AWS S3
Icon
Implementing a filesystem in Hadoop
Icon
Implementing an S3 native filesystem in Hadoop
Icon
Summary
10
10. HDFS Federation
Icon
10. HDFS Federation
Icon
Limitations of the older HDFS architecture
Icon
Architecture of HDFS Federation
Icon
HDFS high availability
Icon
HDFS block placement
Icon
Summary
11
11. Hadoop Security
Icon
11. Hadoop Security
Icon
The security pillars
Icon
Authentication in Hadoop
Icon
Authorization in Hadoop
Icon
Data confidentiality in Hadoop
Icon
Audit logging in Hadoop
Icon
Summary
12
12. Analytics Using Hadoop
Icon
12. Analytics Using Hadoop
Icon
Data analytics workflow
Icon
Machine learning
Icon
Apache Mahout
Icon
Document analysis using Hadoop and Mahout
Icon
RHadoop
Icon
Summary
13
A. Hadoop for Microsoft Windows
Icon
A. Hadoop for Microsoft Windows
Icon
Deploying Hadoop on Microsoft Windows
Icon
Summary
14
Index
Icon
Index

Implementing a filesystem in Hadoop

Based on the situation, it might be a necessity to replace HDFS with a filesystem of your choice. Hadoop provides out-of-the-box support for a few filesystems such as S3. HDFS replacement can be done either as a drop-in replacement or, as in the case with S3, seamless integration with the S3 file store for input and output.

In this section, we will re-implement the S3 native filesystem and extend Hadoop. The code in this section illustrates the steps on how HDFS replacement can be done. Error handling and other features related to S3 have been omitted for brevity.

The major steps in implementing a filesystem for Hadoop are as follows:

  1. The org.apache.hadoop.fs.FileSystem abstract class needs to be extended and all the abstract methods need to be overridden. There are out-of-the-box implementations for FilterFileSystem, NativeS3FileSystem, S3FileSystem, RawLocalFileSystem, FTPFileSystem, and ViewFileSystem.
  2. The open method returns an FsDataInputStream object...
CONTINUE READING
83
Tech Concepts
36
Programming languages
73
Tech Tools
Icon Unlimited access to the largest independent learning library in tech of over 8,000 expert-authored tech books and videos.
Icon Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.
Icon 50+ new titles added per month and exclusive early access to books as they are being written.
Mastering Hadoop
End of Section 9
Next Section
notes
bookmark Notes and Bookmarks search Search in title playlist Add to playlist font-size Font size

Change the font size

margin-width Margin width

Change margin width

day-mode Day/Sepia/Night Modes

Change background colour

Close icon Search
Country selected
Close icon Your notes and bookmarks

Confirmation

Modal Close icon
claim successful
Order Page Read Now

Buy this book with your credits?

Modal Close icon
Are you sure you want to buy this book with one of your credits?
Close
YES, BUY

Submit Your Feedback

Modal Close icon
Modal Close icon
Modal Close icon