Mesos can act as a cluster manager for Spark. While running Spark on Mesos, Spark leverages all the resource management capabilities of Mesos, and Spark tasks are executed on Mesos worker nodes using the Spark executor. This allows the sharing of resources between multiple instances of Spark or with other frameworks. Let's see how to install Spark on Mesos:
Build and run Mesos, as shown in Chapter 1, Running Mesos.
Download the Spark tar file, which is similar to the steps in the earlier section.
The Spark archive containing executors has to be accessible from Mesos. Typically, we can use Hadoop Distributed File System (HDFS) or Amazon S3. We will use HDFS:
ubuntu@master:~ $ hadoop fs -mkdir /tmp ubuntu@master:~ $ hadoop fs -put spark.tar.gz /tmp
Create
spark-env.sh
fromspark-env.sh.template
, and add the following three export lines to the file:ubuntu@master:~ $ cp spark-env.sh.template spark-env.sh ubuntu@master:~ $ vim spark-env.sh export MESOS_NATIVE_LIBRARY=/usr/local...