Amazon Elastic Compute Cloud (EC2) is a web service that provides resizable computer capacity in the cloud. By using Amazon EC2, we can practice HBase on a fully distributed mode easily, at low cost. All the servers that we will use to demonstrate HBase in this book are running on Amazon EC2.
This recipe describes the setup of the Amazon EC2 environment, as a preparation for the installation of HBase on it. We will set up a name server and client on Amazon EC2. You can also use other hosting services such as Rackspace, or real servers to set up your HBase cluster.
You will need to sign up, or create an Amazon Web Service (AWS) account at http://aws.amazon.com/.
We will use EC2 command-line tools to manage our instances. You can download and set up the tools by following the instructions available at the following page:
http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/index.html?SettingUp_CommandLine.html.
You need a public/private key to log in to your EC2 instances. You can generate your key pairs and upload your public key to EC2, using these instructions:
http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/generating-a-keypair.html.
Before you can log in to an instance, you must authorize access. The following link contains instructions for adding rules to the default security group:
http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/adding-security-group-rules.html.
After all these steps are done, review the following checklist to make sure everything is ready:
X.509 certificates: Check if the X.509 certificates are uploaded. You can check this at your account's Security Credentials page.
EC2 key pairs: Check if EC2 key pairs are uploaded. You can check this at AWS Management Console | Amazon EC2 | NETWORK & SECURITY | Key Pairs.
Access: Check if the access has been authorized. This can be checked at AWS Management Console | Amazon EC2 | NETWORK & SECURITY | Security Groups | Inbound.
Environment variable settings: Check if the environment variable settings are done. As an example, the following snippet shows my settings; make sure you are using the right
EC2_URLfor your region:
$ cat ~/.bashrc
export EC2_HOME=~/opt/ec2-api-tools-1.4.4.2
export PATH=$PATH:$EC2_HOME/bin
export EC2_PRIVATE_KEY=~/.ec2/pk-OWRHNWUG7UXIOPJXLOBC5UZTQBOBCVQY.pem
export EC2_CERT=~/.ec2/cert-OWRHNWUG7UXIOPJXLOBC5UZTQBOBCVQY.pem
export JAVA_HOME=/Library/Java/Home
export EC2_URL=https://ec2.us-west-1.amazonaws.com
We need to import our EC2 key pairs to manage EC2 instances via EC2 command-line tools:
$ ec2-import-keypair your-key-pair-name --public-key-file ~/.ssh/id_rsa.pub
Verify the settings by typing the following command:
$ ec2-describe-instances
If everything has been set up properly, the command will show your instances similarly to how you had configured them in the previous command.
Note
Downloading the example code
You can download the example code files for all Packt books you have purchased from your account at http://www.packtpub.com. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you.
The last preparation is to find a suitable AMI. An AMI is a preconfigured operating system and software, which is used to create a virtual machine within EC2. We can find a registered Debian AMI at http://wiki.debian.org/Cloud/AmazonEC2Image.
For the purpose of practicing HBase, a 32-bit, EBS-backed AMI is the most cost effective AMI to use. Make sure you are choosing AMIs for your region. As we are using US-West (us-west-1) for this book, the AMI ID for us is ami-77287b32. This is a 32-bit, small instance of EC2. A small instance is good for practicing HBase on EC2 because it's cheap. For production, we recommend you to use at least High-Memory Extra Large Instance with EBS, or a real server.
Follow these instructions to get your EC2 instances ready for HBase. We will start two EC2 instances; one is a DNS/NTP server, and the other one is the client:
1. Start a micro instance for the mentioned server. We will use
ns1.hbase-admin-cookbook.com (ns1)as its Fully Qualified Domain Name (FQDN) , in a later section of this book:$ ec2-run-instances ami-77287b32 -t t1.micro -k your-key-pair2. Start a small instance for the client. We will use
client1.hbase-admin-cookbook.com (client1)as its FQDN, later in this book:$ ec2-run-instances ami-77287b32 -t m1.small -k your-key-pair3. Verify the startup from AWS Management Console, or by typing the following command:
$ ec2-describe-instancesYou should see two instances from the output of the command. From the output of the
ec2-describe-instancescommand, or AWS Management Console, you can find the public DNS of the instances that have already started. The DNS shows a value such asec2-xx-xx-xxx-xx.us-west-1.compute.amazonaws.com:
4. Log in to the instances via
SSH, using the following command:$ ssh [email protected]5. Update the package index files before we install packages on the server, by using the following command:
root# apt-get update6. Change your instances' time zone to your local timezone, using the following command:
root# dpkg-reconfigure tzdata7. Install the NTP server daemon on the DNS server, using the following command:
root@ns# apt-get install ntp ntp-server ntpdate8. Install the NTP client on the client/server, using the following command:
root@client1# apt-get install ntp ntpdate9. Configure
/etc/ntp.confonns1to run as an NTP server, andclient1to run as an NTP client, usingns1as its server.Because there is no HBase-specific configuration for the NTP setup, we will skip the details. You can find the sample
ntp.conffiles for both the server and client, from the sample source of this book.10. Install BIND9 on
ns1to run as a DNS server, using the following command:root@ns# apt-get install bind9You will need to configure BIND9 to run as a primary master server for internal lookup, and run as a caching server for external lookup. You also need to configure the DNS server, to allow other EC2 instances to update their record on the DNS server.
We will skip the details as this is out of the scope of this book. For sample BIND9 configuration, please refer to the source, shipped with this book.
11. For
client1, just set it up usingns1as its DNS server:root@client1# vi /etc/resolv.conf nameserver 10.160.49.250 #private IP of ns search hbase-admin-cookbook.com #domain name12. Update the DNS hostname automatically. Set up hostname to the EC2 instance's user data of the client. From the My Instances page of AWS Management Console, select
client1from the instances list, stop it, and then click Instance Actions | View | Change User Data; enter thehostnameof the instance you want to use (hereclient1)in the pop-up page:
13. Create a script to update the client's record on the DNS server, using user data:
root@client1# vi ec2-hostname.sh #!/bin/bash #you will need to set up your DNS server to allow update from this key DNS_KEY=/root/etc/Kuser.hbase-admin-cookbook.com.+157+44141.private DOMAIN=hbase-admin-cookbook.com USER_DATA=`/usr/bin/curl -s http://169.254.169.254/latest/user-data` HOSTNAME=`echo $USER_DATA` #set also the hostname to the running instance hostname $HOSTNAME #we only need to update for local IP LOCIP=`/usr/bin/curl -s http://169.254.169.254/latest/meta-data/local-ipv4` cat<<EOF | /usr/bin/nsupdate -k $DNS_KEY -v server ns.$DOMAIN zone $DOMAIN update delete $HOSTNAME.$DOMAIN A update add $HOSTNAME.$DOMAIN 60 A $LOCIP send EOF14. Finally, to run this at boot time from
rc.local, add the following script to therc.localfile:root@client1# vi /etc/rc.local sh /root/bin/ec2-hostname.sh
First we started two instances, a micro instance for DNS/NTP server, and a small one for client. To provide a name service to other instances, the DNS name server has to be kept running. Using micro instance can reduce your EC2 cost.
In step 3, we set up the NTP server and client. We will run our own NTP server on the same DNS server, and NTP clients on all other servers.
EC2 instances can be started and stopped on demand; we don't need to pay for stopped instances. But, restarting an EC2 instance will change the IP address of the instance, which makes it difficult to run HBase. We can resolve this issue by running a DNS server to provide a name service to all EC2 instances in our HBase cluster. We can update name records on the DNS server every time other EC2 instances are restarted.
That's exactly what we have done in steps 4 and 5. Step 4 is a normal DNS setup. In step 5, we stored the instance name in its user data property at first, so that when the instance is restarted, we can get it back using EC2 API. Also, we will get the private IP address of the instance via EC2 API. With this data, we can then send a DNS update command to our DNS server every time the instance is restarted. As a result, we can always use its fixed hostname to access the instance.
We will keep only the DNS instance running constantly. You can stop all other instances whenever you do not need to run your HBase cluster.