Book Image

Learning YARN

Book Image

Learning YARN

Overview of this book

Today enterprises generate huge volumes of data. In order to provide effective services and to make smarter and more intelligent decisions from these huge volumes of data, enterprises use big-data analytics. In recent years, Hadoop has been used for massive data storage and efficient distributed processing of data. The Yet Another Resource Negotiator (YARN) framework solves the design problems related to resource management faced by the Hadoop 1.x framework by providing a more scalable, efficient, flexible, and highly available resource management framework for distributed data processing. This book starts with an overview of the YARN features and explains how YARN provides a business solution for growing big data needs. You will learn to provision and manage single, as well as multi-node, Hadoop-YARN clusters in the easiest way. You will walk through the YARN administration, life cycle management, application execution, REST APIs, schedulers, security framework and so on. You will gain insights about the YARN components and features such as ResourceManager, NodeManager, ApplicationMaster, Container, Timeline Server, High Availability, Resource Localisation and so on. The book explains Hadoop-YARN commands and the configurations of components and explores topics such as High Availability, Resource Localization and Log aggregation. You will then be ready to develop your own ApplicationMaster and execute it over a Hadoop-YARN cluster. Towards the end of the book, you will learn about the security architecture and integration of YARN with big data technologies like Spark and Storm. This book promises conceptual as well as practical knowledge of resource management using YARN.
Table of Contents (20 chapters)
Learning YARN
Credits
About the Authors
Acknowledgments
About the Reviewers
www.PacktPub.com
Preface
Index

The integration of Spark with YARN


Spark is a distributed computing framework that uses in-memory primitives to process data available in a data store. It provides an in-memory representation of data to be processed and it is well suited for various machine learning algorithms. Spark allows easy connection to different data stores such as HDFS, Cassandra, and Amazon S3.

There are several companies that use Spark for big data processing. The complete list of companies and their use cases is available at https://cwiki.apache.org/confluence/display/SPARK/Powered+By+Spark.

Spark has two components: SparkContext (Driver) and Executor. SparkContext is a master service that connects with a cluster manager and acquires resources for Executor services on worker nodes. For cluster management, Spark supports YARN, Apache Mesos and an in-built standalone cluster manager.

In this section, we'll discuss how Spark is integrated with YARN and how you can submit Spark-YARN applications on a Hadoop-YARN cluster...