The Amazon Elastic Compute Cloud (EC2) provides access to compute the capacity in a pay-as-you-go model through virtual machines and is an excellent way of trying out Mesos. Mesos provides scripts to create Mesos clusters of various configurations on EC2. The mesos-ec2 script located in the ec2 directory allows launching, running jobs, and tearing down the Mesos clusters. Note that we can use this script even without building Mesos, but you will need Python (>=2.6). We can manage multiple clusters using different names.
We will need an AWS keypair to use the ec2 script, and our access and secret key. We have to make our keys available via an environment variable. Create and download a keypair via the AWS Management Console (https://console.aws.amazon.com/console/home) and give them 600 permissions:
ubuntu@local:~ $ chmod 600 my-aws-key.pem ubuntu@local:~ $ export AWS_ACCESS_KEY_ID=<your-access-key> ubuntu@local:~ $ export AWS_SECRET_ACCESS_KEY=<your-secret-key>
Now we can use the EC2 scripts provided with Mesos to launch a new cluster using the following command:
ubuntu@local:~/mesos/ec2 $ ./mesos-ec2 -k <your-key-pair> -i <your-identity-file> -s 3 launch ec2-test
This will launch a cluster named ec2-test with three slaves. Once the scripts are done, it will also print the Mesos web UI link, in the form of <master-hostname>:8080. We can confirm that the cluster is up by going to the web interface. The script provides a number of options, a few of which are listed in the following table. We can list all the available options of the script by running mesos-ec2 --help:
|
Command |
Use |
|---|---|
--slave or –s |
This is the number of slaves in the cluster |
--key-pair or -k |
This is the SSH keypair for authentication |
--identity-file or –i |
This is the SSH identity file used for logging into the instances |
--instance-type or –t |
This is a slave instance type, must be 64-bit |
--ebs-vol-size |
This is the size of an EBS volume used to store the persistent HDFS data. |
--master-instance-type or –m |
This is a master instance type, must be 64-bit |
--zone or -z |
This is the Amazon availability zone for launching instances |
--resume |
This flag resumes the installation from the previous run |
We can use the login action to log in to the launched cluster by providing a cluster name, as follows:
ubuntu@local:~/mesos/ec2 $ ./mesos-ec2 -k <your-key-pair> -i <your-identity-file> login ec2-test
The script also sets up a HDFS instance that can be used via commands in the /root/ephemeral-hdfs/ directory.
Finally, we can terminate a cluster using the following command. Be sure to copy any important data before terminating the cluster:
ubuntu@local:~/mesos/ec2 $ ./mesos-ec2 destroy ec2-test
The script also supports advance functionalities, such as pausing and restarting clusters with EBS-backed instances. The Mesos documentation is a great source of information for any clarification. It is worth mentioning that Mesosphere (http://mesosphere.com) also provides you with an easy way of creating an elastic Mesos cluster on Amazon EC2, Google Cloud, and other platforms and provides commercial support for Mesos.