Book Image

YARN Essentials

By : Fasale, Nirmal Kumar
Book Image

YARN Essentials

By: Fasale, Nirmal Kumar

Overview of this book

If you have a working knowledge of Hadoop 1.x but want to start afresh with YARN, this book is ideal for you. You will be able to install and administer a YARN cluster and also discover the configuration settings to fine-tune your cluster both in terms of performance and scalability. This book will help you develop, deploy, and run multiple applications/frameworks on the same shared YARN cluster.
Table of Contents (12 chapters)
Free Chapter
1
1. Need for YARN
9
9. YARN – Alternative Solutions
11
Index

YARN's MapReduce support


MapReduce was the only use case on which the previous versions of Hadoop were developed. We know that MapReduce is mainly used for the efficient and effective processing of big data. It is used to process a graph and millions of its nodes and edges. Going forward with technology, to cater for the requirements of data location availability, fault tolerant systems, and application priorities, YARN built support for everything from a simple shell script application to a complex MapReduce application.

For the data location availability, MapReducer's ApplicationMaster has to find out the data block locations and allocations of containers to process these blocks accordingly. Fault tolerant system means the ability to handle failed tasks and act on them accordingly, such as to handle failed map and reduce tasks and rerun them with other containers if needed. Priorities are assigned to each application in the queue; the logic to handle complex intra-application priorities...