Book Image

Apache Hive Cookbook

Book Image

Apache Hive Cookbook

Overview of this book

Hive was developed by Facebook and later open sourced in Apache community. Hive provides SQL like interface to run queries on Big Data frameworks. Hive provides SQL like syntax also called as HiveQL that includes all SQL capabilities like analytical functions which are the need of the hour in today’s Big Data world. This book provides you easy installation steps with different types of metastores supported by Hive. This book has simple and easy to learn recipes for configuring Hive clients and services. You would also learn different Hive optimizations including Partitions and Bucketing. The book also covers the source code explanation of latest Hive version. Hive Query Language is being used by other frameworks including spark. Towards the end you will cover integration of Hive with these frameworks.
Table of Contents (19 chapters)
Apache Hive Cookbook
Credits
About the Authors
About the Reviewer
www.PacktPub.com
Preface
Index

Installing Hive


We will now take a look at installing Hive along with all the prerequisites.

Getting ready

Let's download the stable version from one of the mirrors:

$ wget http://a.mbbsindia.com/hive/hive-1.2.1/apache-hive-1.2.1-bin.tar.gz

How to do it…

This can be achieved in three ways.

Hive with an embedded metastore

Once you have downloaded the Hive tar-ball file, installing and setting up a Hive is pretty simple and straightforward. Extract the compressed tar:

$tar –xzvf apache-hive-1.2.1-bin.tar.gz

Export the location where Hive is extracted as the environment variable HIVE_HOME:

$ cd  apache-hive-1.2.1-bin
$ export HIVE_HOME={{pwd}}

Hive has all its installation scripts in the $HIVE_HOME/bin directory. Export this location to the PATH environment variable so that you can run all scripts from any location directly from a command-line:

$ export PATH=$HIVE_HOME/bin:$PATH

Alternatively, if you want to set the Hive path permanently for the user, then make the entry of Hive environment variables in the .bashrc or .bash_profile files available or could be created in the user's home folder:

  1. Add the following to ~/.bash_profile:

    export HIVE_HOME=/home/hduser/apache-hive-1.2.1-bin
    export PATH=$PATH:$HIVE_HOME/bin
    
  2. Here, hduser is the name of user with which you have logged in and Hive-1.2.1 is the Hive directory extracted from the tar file.
Run Hive from a terminal:

    hive
    
  3. Make sure that the Hive node has a connection to Hadoop cluster, which means Hive would be installed on any of the Hadoop nodes, or Hadoop configurations are available in the node's class path.

  4. This installation uses the embedded Derby database and stores the data on the local filesystem. Only one Hive session can be open on the node.

  5. If different users try to run the Hive shell, the second would get the Failed to start database 'metastore_db' error.

  6. Run Hive queries for the datastore to test the installation:

    hive> SHOW TABLES;
    hive> CREATE TABLE sales(id INT, product String, age INT) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t';
    
  7. Logs are generated per user bases in the /tmp/<usrename> folder.

Hive with a local metastore

Follow these steps to configure Hive with the local metastore. Here, we are using the MySQL database as a metastore:

  1. Add following to ~/.bash_profile:

    export HIVE_HOME=/home/hduser/apache-hive-1.2.1-bin
    export PATH=$PATH:$HIVE_HOME/bin
    

    Here, hduser is the user name, and apache-hive-1.2.1-bin is the Hive directory extracted from the tar file.

  2. Install a SQL database such as MySQL on the same machine where you want to run Hive.

  3. For the Ubuntu, MySQL could be installed by running the following command on the node's terminal:

    sudo apt-get install mysql-server
    
  4. In case of MySql, Hive needs the mysql-connector jar. Download the latest mysql-connector jar from http://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.35.tar.gz and copy it to the lib folder of your Hive home directory.

  5. Create a file, hive-site.xml, in the conf folder of Hive and add the following entries to it:

    <configuration>
    <property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://localhost:3306/metastore_db?createDatabaseIfNotExist=true</value>
    <description>metadata is stored in a MySQL server</description>
    </property>
    <property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>com.mysql.jdbc.Driver</value>
    <description>MySQL JDBC driver class</description>
    </property>
    <property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>hduser</value>
    <description>user name for connecting to mysql server     
    </description>
    </property>
    <property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>passwd</value>
    <description>password for connecting to mysql server</description>
    </property>
    </configuration>
  6. Run Hive from the terminal:

    hive
    

Note

There is a known "JLine" jar conflict issue with Hadoop 2.6.0 and Hive 1.2.1. If you are getting the error "unable to load class jline.terminal," you need to remove the older version of the jline jar from the yarn lib folder using the following command:

sudo rm -r $HADOOP_PREFIX/share/hadoop/yarn/lib/jline-0.9.94.jar

Hive with a remote metastore

Follow these steps to configure Hive with a remote metastore.

  1. Download the latest version of Hive from http://a.mbbsindia.com/hive/hive-1.2.1/apache-hive-1.2.1-bin.tar.gz.

  2. Extract the package:

    tar –xzvf apache-hive-1.2.1-bin.tar.gz
    
  3. Add the following to ~/.bash_profile:

    sudo nano ~/.bash_profile
    export HIVE_HOME=/home/hduser/apache-hive-1.2.1-bin
    export PATH=$PATH:$HIVE_HOME/bin
    

    Here, hduser is the user name and apache-hive-1.2.1-bin is the Hive directory extracted from the tar file.

  4. Install a SQL database such as MySQL on a remote machine to be used for the metastore.

  5. For Ubuntu, MySQL can be installed with the following command:

    sudo apt-get install mysql-server
    
  6. In the case of MySQL, Hive needs the mysql-connector jar file. Download the latest mysql-connector jar from http://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.35.tar.gz and copy it to the lib folder of your Hive home directory.

  7. Add the following entries to hive-site.xml:

    <configuration>
    <property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://<ip_of_remote_host>:3306/metastore_db?createDatabaseIfNotExist=true</value>
    <description>metadata is stored in a MySQL server</description>
    </property>
    <property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>com.mysql.jdbc.Driver</value><description>MySQL JDBC driver class</description>
    </property>
    <property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>hduser</value>
    <description>user name for connecting to mysql server     
    </description>
    </property>
    <property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>passwd</value>
    <description>password for connecting to mysql server</description>
    </property>
    </configuration>
  8. Start the Hive metastore interface:

    bin/hive --service metastore &
    
  9. Run Hive from the terminal:

    hive
    
  10. The Hive metastore interface by default listens at port 9083:

    netstat -an | grep 9083
    
  11. Start the Hive shell and make sure that the Hive Data Definition Language and Data Manipulation Language (DDL or DML) operations are working by creating tables in Hive.

Note

There is a known "JLine" jar conflict issue with Hadoop 2.6.0 and Hive 1.2.1. If you are getting the error "unable to load class jline.terminal," you need to remove the older version of jline jar from the yarn lib folder using the following command:

sudo rm -r $HADOOP_PREFIX/share/hadoop/yarn/lib/jline-0.9.94.jar