Book Image

Mastering Hadoop

By : Sandeep Karanth
Book Image

Mastering Hadoop

By: Sandeep Karanth

Overview of this book

Table of Contents (21 chapters)
Mastering Hadoop
About the Author
About the Reviewers

Chapter 6. YARN – Bringing Other Paradigms to Hadoop

Yet Another Resource Negotiator (YARN) is a cluster resource management layer that was introduced in Hadoop 2.0. As we saw briefly in Chapter 1, Hadoop 2.X, YARN separates out the responsibilities of the JobTracker daemon. JobTracker was responsible for:

  • Resource arbitration within a Hadoop cluster

  • MapReduce job management

The problem with the JobTracker model was that it became the single point of failure in the compute layer of a Hadoop cluster. Any failure in JobTracker meant trashing the running jobs and starting all over again. JobTracker's singular nature also became a scaling bottleneck. All job communications, scheduling, and resource management were controlled by the JobTracker master daemon.

The tightly coupled functions of JobTracker made it rigid, allowing a single computing paradigm, MapReduce, to be onboarded onto the cluster. MapReduce is not suitable for a variety of emerging applications and force-fitting solutions to all...