Browse Library
Sign In Start Free Trial
Close icon Search icon
Account
Browse Library Advanced Search Sign In Start Free Trial

Add to playlist

Create a Playlist

Modal Close icon
You need to login to use this feature.
  • Book Overview & Buying Learning YARN
  • Table Of Contents Toc
  • Feedback & Rating feedback
Learning YARN

Learning YARN

By : Akhil Arora, Shrey Mehrotra
4 (2)
Buy this Book
close
close
Learning YARN

Learning YARN

4 (2)
By: Akhil Arora, Shrey Mehrotra
Buy this Book

Overview of this book

Today enterprises generate huge volumes of data. In order to provide effective services and to make smarter and more intelligent decisions from these huge volumes of data, enterprises use big-data analytics. In recent years, Hadoop has been used for massive data storage and efficient distributed processing of data. The Yet Another Resource Negotiator (YARN) framework solves the design problems related to resource management faced by the Hadoop 1.x framework by providing a more scalable, efficient, flexible, and highly available resource management framework for distributed data processing. This book starts with an overview of the YARN features and explains how YARN provides a business solution for growing big data needs. You will learn to provision and manage single, as well as multi-node, Hadoop-YARN clusters in the easiest way. You will walk through the YARN administration, life cycle management, application execution, REST APIs, schedulers, security framework and so on. You will gain insights about the YARN components and features such as ResourceManager, NodeManager, ApplicationMaster, Container, Timeline Server, High Availability, Resource Localisation and so on. The book explains Hadoop-YARN commands and the configurations of components and explores topics such as High Availability, Resource Localization and Log aggregation. You will then be ready to develop your own ApplicationMaster and execute it over a Hadoop-YARN cluster. Towards the end of the book, you will learn about the security architecture and integration of YARN with big data technologies like Spark and Storm. This book promises conceptual as well as practical knowledge of resource management using YARN.
Table of Contents (14 chapters)
close
close
close
Preface
Icon
Preface
Icon
What this book covers
Icon
What you need for this book
Icon
Who this book is for
Icon
Conventions
Icon
Reader feedback
Icon
Customer support
Lock Free Chapter
1
1. Starting with YARN Basics
chevron up
Icon
1. Starting with YARN Basics
Icon
Introduction to MapReduce v1
Icon
Shortcomings of MapReducev1
Icon
An overview of YARN components
Icon
The YARN architecture
Icon
How YARN satisfies big data needs
Icon
Projects powered by YARN
Icon
Summary
2
2. Setting up a Hadoop-YARN Cluster
Icon
2. Setting up a Hadoop-YARN Cluster
Icon
Starting with the basics
Icon
The Hadoop-YARN single node installation
Icon
An overview of web user interfaces
Icon
The Hadoop-YARN multi-node installation
Icon
An overview of the Hortonworks and Cloudera installations
Icon
Summary
3
3. Administering a Hadoop-YARN Cluster
Icon
3. Administering a Hadoop-YARN Cluster
Icon
Using the Hadoop-YARN commands
Icon
Configuring the Hadoop-YARN services
Icon
Managing the Hadoop-YARN services
Icon
Monitoring the YARN services
Icon
Understanding ResourceManager's High Availability
Icon
Monitoring NodeManager's health
Icon
Summary
4
4. Executing Applications Using YARN
Icon
4. Executing Applications Using YARN
Icon
Understanding application execution flow
Icon
Submitting a sample MapReduce application
Icon
Handling failures in YARN
Icon
YARN application logging
Icon
Summary
5
5. Understanding YARN Life Cycle Management
Icon
5. Understanding YARN Life Cycle Management
Icon
An introduction to state management analogy
Icon
The ResourceManager's view
Icon
The NodeManager's view
Icon
Analyzing transitions through logs
Icon
Summary
6
6. Migrating from MRv1 to MRv2
Icon
6. Migrating from MRv1 to MRv2
Icon
Introducing MRv1 and MRv2
Icon
High-level changes from MRv1 to MRv2
Icon
The migration steps from MRv1 to MRv2
Icon
Running and monitoring MRv1 apps on YARN
Icon
Summary
7
7. Writing Your Own YARN Applications
Icon
7. Writing Your Own YARN Applications
Icon
An introduction to the YARN API
Icon
Writing your own application
Icon
Summary
8
8. Dive Deep into YARN Components
Icon
8. Dive Deep into YARN Components
Icon
Understanding ResourceManager
Icon
Understanding NodeManager
Icon
The YARN Timeline server
Icon
The web application proxy server
Icon
YARN Scheduler Load Simulator (SLS)
Icon
Handling resource localization in YARN
Icon
Summary
9
9. Exploring YARN REST Services
Icon
9. Exploring YARN REST Services
Icon
Introduction to YARN REST services
Icon
ResourceManager REST APIs
Icon
NodeManager REST APIs
Icon
MapReduce ApplicationMaster REST APIs
Icon
MapReduce HistoryServer REST APIs
Icon
How to access REST services
Icon
Summary
10
10. Scheduling YARN Applications
Icon
10. Scheduling YARN Applications
Icon
An introduction to scheduling in YARN
Icon
An overview of queues
Icon
Types of queues
Icon
An introduction to schedulers
Icon
Summary
11
11. Enabling Security in YARN
Icon
11. Enabling Security in YARN
Icon
Adding security to a YARN cluster
Icon
Working with ACLs
Icon
Other security frameworks
Icon
Summary
12
12. Real-time Data Analytics Using YARN
Icon
12. Real-time Data Analytics Using YARN
Icon
The integration of Spark with YARN
Icon
The integration of Storm with YARN
Icon
The integration of HAMA and Giraph with YARN
Icon
Summary
13
Index
Icon
Index

Chapter 1. Starting with YARN Basics

In early 2006, Apache Hadoop was introduced as a framework for the distributed processing of large datasets stored across clusters of computers, using a programming model. Hadoop was developed as a solution to handle big data in a cost effective and easiest way possible. Hadoop consisted of a storage layer, that is, Hadoop Distributed File System (HDFS) and the MapReduce framework for managing resource utilization and job execution on a cluster. With the ability to deliver high performance parallel data analysis and to work with commodity hardware, Hadoop is used for big data analysis and batch processing of historical data through MapReduce programming.

With the exponential increase in the usage of social networking sites such as Facebook, Twitter, and LinkedIn and e-commerce sites such as Amazon, there was the need of a framework to support not only MapReduce batch processing, but real-time and interactive data analysis as well. Enterprises should be able to execute other applications over the cluster to ensure that cluster capabilities are utilized to the fullest. The data storage framework of Hadoop was able to counter the growing data size, but resource management became a bottleneck. The resource management framework for Hadoop needed a new design to solve the growing needs of big data.

YARN, an acronym for Yet Another Resource Negotiator, has been introduced as a second-generation resource management framework for Hadoop. YARN is added as a subproject of Apache Hadoop. With MapReduce focusing only on batch processing, YARN is designed to provide a generic processing platform for data stored across a cluster and a robust cluster resource management framework.

In this chapter, we will cover the following topics:

  • Introduction to MapReduce v1
  • Shortcomings of MapReduce v1
  • An overview of the YARN components
  • The YARN architecture
  • How YARN satisfies big data needs
  • Projects powered by YARN
Visually different images
CONTINUE READING
83
Tech Concepts
36
Programming languages
73
Tech Tools
Icon Unlimited access to the largest independent learning library in tech of over 8,000 expert-authored tech books and videos.
Icon Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.
Icon 50+ new titles added per month and exclusive early access to books as they are being written.
Learning YARN
End of Section 1
Next Section
notes
bookmark Notes and Bookmarks search Search in title playlist Add to playlist font-size Font size

Change the font size

margin-width Margin width

Change margin width

day-mode Day/Sepia/Night Modes

Change background colour

Close icon Search
Country selected
Close icon Your notes and bookmarks

Confirmation

Modal Close icon
claim successful
Order Page Read Now

Buy this book with your credits?

Modal Close icon
Are you sure you want to buy this book with one of your credits?
Close
YES, BUY

Submit Your Feedback

Modal Close icon
Modal Close icon
Modal Close icon