Book Image

Apache Hive Essentials. - Second Edition

By : Dayong Du
Book Image

Apache Hive Essentials. - Second Edition

By: Dayong Du

Overview of this book

In this book, we prepare you for your journey into big data by frstly introducing you to backgrounds in the big data domain, alongwith the process of setting up and getting familiar with your Hive working environment. Next, the book guides you through discovering and transforming the values of big data with the help of examples. It also hones your skills in using the Hive language in an effcient manner. Toward the end, the book focuses on advanced topics, such as performance, security, and extensions in Hive, which will guide you on exciting adventures on this worthwhile big data journey. By the end of the book, you will be familiar with Hive and able to work effeciently to find solutions to big data problems
Table of Contents (12 chapters)

Spark

As a general-purpose data engine, Apache Spark can integrate with Hive closely. Spark SQL has supported a subset of HQL and can leverage the Hive metastore to write or query data in Hive. This approach is also called Spark over Hive. To configure Spark, use Hive the metastore, you only need to copy the hive-site.xml to the ${SPARK_HOME}/conf directory. After that, running the spark-sql command will enter the Spark SQL interactive environment, where you can write SQL to query Hive tables.

On the other hand, Hive over Spark is a similar approach, but lets Hive use Spark as an alternative engine. In this case, users still stay in Hive and write HQL, but run over the Spark engine transparently. Hive over Spark requires the Yarn FairScheduler and set hive.execution.engine=spark. For more details, refer to https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting...