Book Image

YARN Essentials

Book Image

YARN Essentials

Overview of this book

Table of Contents (17 chapters)
YARN Essentials
About the Authors
About the Reviewers
Free Chapter
Need for YARN
YARN – Alternative Solutions


Omega is Google's next generation cluster management system.

Omega is specifically focused on a cluster scheduling architecture that uses parallelism, shared state, and optimistic concurrency control.

From the past experience, Google noticed that as the clusters and their workloads increase, the scheduler is at risk of becoming a scalability bottleneck.

Google's production job scheduler has experienced all of this. Over the years, it has evolved into a complicated, sophisticated system that is hard to change.

A schematic overview of the scheduling architectures can be seen in the following figure:

  • contrib project to Hadoop 0.20 branch and is not a very large code base.

  • Corona is integrated with the fair-scheduler.

  • YARN is more interested in the capacity scheduler.

Google identified the following two prevalent scheduler architectures shown in the preceding figure:

  • Monolithic schedulers: This uses a single, centralized scheduling algorithm for all jobs (our existing scheduler is one of these...