Mesos is slowly emerging as a data center operating system to manage all compute resources across a data center. Mesos runs on any computer running the Linux operating system. Mesos is built using the same principles as Linux kernel. Let's see how we can install Mesos.
Mesosphere provides a binary distribution of Mesos. The most recent package for the Mesos distribution can be installed from the Mesosphere repositories by performing the following steps:
Execute Mesos on Ubuntu OS with the trusty version:
$ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv E56151BF DISTRO=$(lsb_release -is | tr '[:upper:]' '[:lower:]') CODENAME=$(lsb_release -cs) $ sudo vi /etc/apt/sources.list.d/mesosphere.list deb http://repos.mesosphere.io/Ubuntu trusty main
Update the repositories:
$ sudo apt-get -y updateInstall Mesos:
$ sudo apt-get -y install mesosTo connect Spark to Mesos to integrate Spark with Mesos, make Spark binaries available to Mesos and configure the Spark driver to connect to Mesos.
Use Spark binaries from the first recipe and upload to HDFS:
$ hdfs dfs -put spark-1.4.0-bin-hadoop2.4.tgz spark-1.4.0-bin-hadoop2.4.tgz
The master URL for single master Mesos is
mesos://host:5050, and for the ZooKeeper managed Mesos cluster, it ismesos://zk://host:2181.Set the following variables in
spark-env.sh:$ sudo vi spark-env.sh export MESOS_NATIVE_LIBRARY=/usr/local/lib/libmesos.so export SPARK_EXECUTOR_URI= hdfs://localhost:9000/user/hduser/spark-1.4.0-bin-hadoop2.4.tgz
Run from the Scala program:
val conf = new SparkConf().setMaster("mesos://host:5050") val sparkContext = new SparkContext(conf)
$ spark-shell --master mesos://host:5050To run in the coarse-grained mode, set the
spark.mesos.coarseproperty:conf.set("spark.mesos.coarse","true")