Yet Another Resource Negotiator (YARN) is a cluster resource management layer that was introduced in Hadoop 2.0. As we saw briefly in Chapter 1, Hadoop 2.X, YARN separates out the responsibilities of the JobTracker daemon. JobTracker was responsible for:
Resource arbitration within a Hadoop cluster
MapReduce job management
The problem with the JobTracker model was that it became the single point of failure in the compute layer of a Hadoop cluster. Any failure in JobTracker meant trashing the running jobs and starting all over again. JobTracker's singular nature also became a scaling bottleneck. All job communications, scheduling, and resource management were controlled by the JobTracker master daemon.
The tightly coupled functions of JobTracker made it rigid, allowing a single computing paradigm, MapReduce, to be onboarded onto the cluster. MapReduce is not suitable for a variety of emerging applications and force-fitting solutions to all...