Starting a Hadoop cluster with the new AMI is simple and straightforward. This recipe will list steps to start up a Hadoop cluster with the new AMI.
Before getting started, we assume that you have registered with AWS and have successfully created a new AMI with Hadoop properly configured.
Use the following steps to configure a Hadoop cluster with EC2:
Run a number of instances either from the command line or from the web interface.
After the instances are all in running state, run the following command to get the internal hostname of these instances:
ec2-describe-instances | grep running | egrep -o 'ip.*?internal' | sed -e 's/.ec2.internal//g' > nodes.txt
The
nodes.txt
file will have contents similar to the following:ip-10-190-81-210 ip-10-137-11-196 ip-10-151-11-161 ip-10-137-48-163 ip-10-143-160-5 ip-10-142-132-17
We are assuming to use the
ip-10-190-81-210
node as the master node and the public domain name of this...