YARN started out as part of the MapReduce v2 (MRv2) initiative but is now an independent sub-project within Hadoop (that is, it's at the same level as MapReduce). It grew out of a realization that MapReduce in Hadoop 1 conflated two related but distinct responsibilities: resource management and application execution.

Although it has enabled previously unimagined processing on enormous datasets, the MapReduce model at a conceptual level has an impact on performance and scalability. Implicit in the MapReduce model is that any application can only be composed of a series of largely linear MapReduce jobs, each of which follows a model of one or more maps followed by one or more reduces. This model is a great fit for some applications, but not all. In particular, it's a poor fit for workloads requiring very low-latency response times; the MapReduce startup times and sometimes lengthy job chains often greatly exceed the tolerance for a user-facing process. The model has also been found to...