Amazon Elastic Compute Cloud (Amazon EC2) is a web service that provides resizable compute instances in the cloud. Amazon EC2 provides the following features:
- On-demand delivery of IT resources via the Internet
- Provisioning of as many instances as you like
- Payment for the hours during which you use instances, such as your utility bill
- No setup cost, no installation, and no overhead at all
- Shutting down or terminating instances when you no longer need them
- Making such instances available on familiar operating systems
EC2 provides different types of instances to meet all your compute needs, such as general-purpose instances, microinstances, memory-optimized instances, storage-optimized instances, and others. They also have a Free Tier of microinstances for trial purposes.
The spark-ec2 script comes bundled with Spark and makes it easy to launch, manage, and shut down clusters on Amazon EC2.
Before you start, do the following things: log in to the Amazon AWS account via http://aws.amazon.com.
- Click on
Security Credentialsunder your account name in the top-right corner. - Click on
Access KeysandCreate New Access Key:
- Download the key file (let's save it in the
/home/hduser/kpfolder asspark-kp1.pem). - Set permissions on the key file to
600. - Set environment variables to reflect
access key IDandsecret access key(replace the sample values with your own values):
$ echo "export AWS_ACCESS_KEY_ID="AKIAOD7M2LOWATFXFKQ"" >> /home/hduser/.bashrc $ echo "export AWS_SECRET_ACCESS_KEY="+Xr4UroVYJxiLiY8DLT4DLT4D4sxc3ijZGMx1D3pfZ2q"" >> /home/hduser/.bashrc $ echo "export PATH=$PATH:/opt/infoobjects/spark/ec2" >> /home/hduser/.bashrc
- Spark comes bundled with scripts to launch the Spark cluster on Amazon EC2. Let's launch the cluster using the following command:
$ cd /home/hduser $ spark-ec2 -k <key-pair> -i <key-file> -s <num-slaves> launch <cluster-name> <key-pair> - name of EC2 keypair created in AWS <key-file> the private key file you downloaded <num-slaves> number of slave nodes to launch <cluster-name> name of the cluster
- Launch the cluster with the example value:
$ spark-ec2 -k kp-spark -i /home/hduser/keypairs/kp-spark.pem --hadoop-major-
version 2 -s 3 launch spark-cluster- Sometimes, the default availability zones are not available; in that case, retry sending the request by specifying the specific availability zone you are requesting:
$ spark-ec2 -k kp-spark -i /home/hduser/keypairs/kp-spark.pem -z us-east-1b --
hadoop-major-version 2 -s 3 launch spark-cluster- If your application needs to retain data after the instance shuts down, attach EBS volume to it (for example, 10 GB space):
$ spark-ec2 -k kp-spark -i /home/hduser/keypairs/kp-spark.pem --hadoop-major-
version 2 -ebs-vol-size 10 -s 3 launch spark-cluster- If you use Amazon's spot instances, here is the way to do it:
$ spark-ec2 -k kp-spark -i /home/hduser/keypairs/kp-spark.pem -spot-price=0.15
--hadoop-major-version 2 -s 3 launch spark-clusterNote
Spot instances allow you to name your own price for Amazon EC2's computing capacity. You simply bid for spare Amazon EC2 instances and run them whenever your bid exceeds the current spot price, which varies in real time and is based on supply and demand (source: www.amazon.com).
- After completing the preceding launch process, check the status of the cluster by going to the webUI URL that will be printed at the end:
- Check the status of the cluster:
- Now, to access the Spark cluster on EC2, connect to the master node using secure shell protocol (SSH):
$ spark-ec2 -k spark-kp1 -i /home/hduser/kp/spark-kp1.pem login spark-cluster- The following image illustrates the result you'll get:
- Check the directories in the master node and see what they do:
Directory | Description |
| This is the Hadoop instance for which data is ephemeral and gets deleted when you stop or restart the machine. |
| Each node has a very small amount of persistent storage (approximately 3 GB). If you use this instance, data will be retained in that space. |
| This refers to the native libraries that support Hadoop, such as snappy compression libraries. |
| This refers to the Scala installation. |
| This refers to the Shark installation (Shark is no longer supported and is replaced by Spark SQL). |
| This refers to the Spark installation. |
| This refers to the files that support this cluster deployment. |
| This refers to the Tachyon installation. |
- Check the HDFS version in an ephemeral instance:
$ ephemeral-hdfs/bin/hadoop version Hadoop 2.0.0-chd4.2.0
- Check the HDFS version in a persistent instance with the following command:
$ persistent-hdfs/bin/hadoop version Hadoop 2.0.0-chd4.2.0
- Change the configuration level of the logs:
$ cd spark/conf- The default log level information is too verbose, so let's change it to Error:
- Create the
log4.propertiesfile by renaming the template:
$ mv log4j.properties.template log4j.properties- Open
log4j.propertiesin vi or your favorite editor:
$ vi log4j.properties- Change the second line from
| log4j.rootCategory=INFO, consoleto| log4j.rootCategory=ERROR, console.
- Copy the configuration to all the slave nodes after the change:
$ spark-ec2/copydir spark/conf- You should get something like this:
- Destroy the Spark cluster:
$ spark-ec2 destroy spark-cluster