Book Image

Apache Hadoop 3 Quick Start Guide

By : Hrishikesh Vijay Karambelkar
Book Image

Apache Hadoop 3 Quick Start Guide

By: Hrishikesh Vijay Karambelkar

Overview of this book

Apache Hadoop is a widely used distributed data platform. It enables large datasets to be efficiently processed instead of using one large computer to store and process the data. This book will get you started with the Hadoop ecosystem, and introduce you to the main technical topics, including MapReduce, YARN, and HDFS. The book begins with an overview of big data and Apache Hadoop. Then, you will set up a pseudo Hadoop development environment and a multi-node enterprise Hadoop cluster. You will see how the parallel programming paradigm, such as MapReduce, can solve many complex data processing problems. The book also covers the important aspects of the big data software development lifecycle, including quality assurance and control, performance, administration, and monitoring. You will then learn about the Hadoop ecosystem, and tools such as Kafka, Sqoop, Flume, Pig, Hive, and HBase. Finally, you will look at advanced topics, including real time streaming using Apache Storm, and data analytics using Apache Spark. By the end of the book, you will be well versed with different configurations of the Hadoop 3 cluster.

Related Content you might be interested in

Current Title:

Small book image
Apache Hadoop 3 Quick Start Guide
Hrishikesh Vijay Karambelkar
Oct, 2018 | 220 pages
Small book image
Mastering Hadoop 3
Manish Kumar | Chanchal Singh
Feb, 2019 | 544 pages
Small book image
Hadoop 2.x Administration Cookbook
Gurmukh Singh
May, 2017 | 348 pages
Small book image
HBase High Performance Cookbook
Ruchir Choudhry
Jan, 2017 | 350 pages
Table of Contents (10 chapters)
Preface
Preface
Who this book is for
What this book covers
To get the most out of this book
Get in touch
Free Chapter
1
Hadoop 3.0 - Background and Introduction
Hadoop 3.0 - Background and Introduction
How it all started
What Hadoop is and why it is important
How Apache Hadoop works
Hadoop 3.0 releases and new features
Choosing the right Hadoop distribution
Summary
2
Planning and Setting Up Hadoop Clusters
Planning and Setting Up Hadoop Clusters
Technical requirements
Prerequisites for Hadoop setup
Running Hadoop in standalone mode
Setting up a pseudo Hadoop cluster
Planning and sizing clusters
Setting up Hadoop in cluster mode
Diagnosing the Hadoop cluster
Summary
3
Deep Dive into the Hadoop Distributed File System
Deep Dive into the Hadoop Distributed File System
Technical requirements
How HDFS works
Key features of HDFS
Data flow patterns of HDFS
HDFS configuration files
Hadoop filesystem CLIs
Working with data structures in HDFS
Summary
4
Developing MapReduce Applications
Developing MapReduce Applications
Technical requirements
How MapReduce works
Configuring a MapReduce environment
Understanding Hadoop APIs and packages
Setting up a MapReduce project
Deep diving into MapReduce APIs
Compiling and running MapReduce jobs
Streaming in MapReduce programming
Summary
5
Building Rich YARN Applications
Building Rich YARN Applications
Technical requirements
Understanding YARN architecture
Key features of YARN
Configuring the YARN environment in a cluster
Working with YARN distributed CLI
Deep dive with YARN application framework
Building and monitoring a YARN application on a cluster
Summary
6
Monitoring and Administration of a Hadoop Cluster
Monitoring and Administration of a Hadoop Cluster
Roles and responsibilities of Hadoop administrators
Planning your distributed cluster
Resource management in Hadoop
High availability of Hadoop
Securing Hadoop clusters
Performing routine tasks
Summary
7
Demystifying Hadoop Ecosystem Components
Demystifying Hadoop Ecosystem Components
Technical requirements
Understanding Hadoop's Ecosystem
Working with Apache Kafka
Writing Apache Pig scripts
Transferring data with Sqoop
Writing Flume jobs
Understanding Hive
Using HBase for NoSQL storage
Summary
8
Advanced Topics in Apache Hadoop
Advanced Topics in Apache Hadoop
Technical requirements
Hadoop use cases in industries
Advanced Hadoop data storage file formats
Real-time streaming with Apache Storm
Data analytics with Apache Spark
Summary
9
Other Books You May Enjoy
Other Books You May Enjoy
Leave a review - let other readers know what you think

Understanding Hadoop APIs and packages

Now let's go through some of the key APIs that you will be using while you program in MapReduce. First, let's understand the important packages that are part of Apache Hadoop MapReduce APIs and their capabilities:

...

Java API Packages

Description

org.apache.Hadoop.mapred

Primarily provides interfaces for MapReduce, input/output formats, and job-related classes. This is an older API.

org.apache.Hadoop.mapred.lib

Contains libraries for Mapper, Reducer, partitioners, and so on. To be avoided—use mapreduce.lib.

org.apache.Hadoop.mapred.pipes

Job submitter-related classes.

org.apache.Hadoop.mapred.tools

Command-line tools associated with MapReduce.

org.apache.Hadoop.mapred.uploader

The org.apache.Hadoop.mapred.uploader package contains classes related to the MapReduce framework upload tool.
Previous Section
End of Section 5
Next Section