Yet Another Resource Negotiator (YARN) is Hadoop's compute framework that runs on top of HDFS, which is Hadoop's storage layer.
YARN follows the master-slave architecture. The master daemon is called ResourceManager and the slave daemon is called NodeManager. Besides this application, life cycle management is done by ApplicationMaster, which can be spawned on any slave node and would be alive during the lifetime of an application.
When Spark is run on YARN, ResourceManager performs the role of the Spark master and NodeManagers works as executor nodes.
While running Spark with YARN, each Spark executor is run as a YARN container.
Running Spark on YARN requires a binary distribution of Spark that has YARN support. In both the Spark installation recipes, we have taken care of this.
- To run Spark on YARN, the first step is to set the configuration:
HADOOP_CONF_DIR: to write to HDFS YARN_CONF_DIR: to connect to YARN ResourceManager $ cd /opt/infoobjects/spark/conf (or /etc/spark) $ sudo vi spark-env.sh export HADOOP_CONF_DIR=/opt/infoobjects/hadoop/etc/Hadoop export YARN_CONF_DIR=/opt/infoobjects/hadoop/etc/hadoop
- You can see this in the following screenshot:
- The following command launches YARN Spark in the
yarn-clientmode:
$ spark-submit --class path.to.your.Class --master yarn --deploy-mode client
[options] <app jar> [app options]Here's an example:
$ spark-submit --class com.infoobjects.TwitterFireHose --master yarn --deploy-
mode client --num-executors 3 --driver-memory 4g --executor-memory 2g --
executor-cores 1 target/sparkio.jar 10- The following command launches
Spark shellin theyarn-clientmode:
$ spark-shell --master yarn --deploy-mode client - The command to launch the spark application in the
yarn-clustermode is as follows:
$ spark-submit --class path.to.your.Class --master yarn --deploy-mode cluster
[options] <app jar> [app options]Here's an example:
$ spark-submit --class com.infoobjects.TwitterFireHose --master yarn --deploy-
mode cluster --num-executors 3 --driver-memory 4g --executor-memory 2g --
executor-cores 1 target/sparkio.jar 10Spark applications on YARN run in two modes:
yarn-client: Spark Driver runs in the client process outside of the YARN cluster, andApplicationMasteris only used to negotiate the resources fromResourceManager.yarn-cluster: Spark Driver runs inApplicationMaster, spawned byNodeManageron a slave node.
The yarn-cluster mode is recommended for production deployments, while the yarn-client mode is good for development and debugging, where you would like to see the immediate output. There is no need to specify the Spark master in either mode as it's picked from the Hadoop configuration, and the master parameter is either yarn-client or yarn-cluster.
The following figure shows how Spark is run with YARN in the client mode:
The following figure shows how Spark is run with YARN in the cluster mode:
In the YARN mode, the following configuration parameters can be set:
--num-executors: To configure how many executors will be allocated--executor-memory: RAM per executor--executor-cores: CPU cores per executor