Book Image

Hadoop MapReduce v2 Cookbook - Second Edition: RAW

Book Image

Hadoop MapReduce v2 Cookbook - Second Edition: RAW

Overview of this book

Table of Contents (19 chapters)
Hadoop MapReduce v2 Cookbook Second Edition
Credits
About the Author
Acknowledgments
About the Author
About the Reviewers
www.PacktPub.com
Preface
Index

Shared user Hadoop clusters – using Fair and Capacity schedulers


The Hadoop YARN scheduler is responsible for assigning resources to the applications submitted by users. In Hadoop YARN, these can be any YARN application in addition to MapReduce applications. Currently, the default YARN resource allocation is based on the memory requirements of the application, while resource allocation based on other resources such as CPU can be configured additionally.

Hadoop YARN supports a pluggable scheduling framework, where the cluster administrator has the choice of selecting an appropriate scheduler for the cluster. By default, YARN supports a First in First out (FIFO) scheduler, which executes jobs in the same order as they arrive using a queue of jobs. However, FIFO scheduling might not be the best option for large multi-user Hadoop deployments, where the cluster resources have to be shared across different users and different applications to ensure maximum productivity from the cluster. Please...