Book Image

Mastering Hadoop

By : Karanth
Book Image

Mastering Hadoop

By: Karanth

Overview of this book

Do you want to broaden your Hadoop skill set and take your knowledge to the next level? Do you wish to enhance your knowledge of Hadoop to solve challenging data processing problems? Are your Hadoop jobs, Pig scripts, or Hive queries not working as fast as you intend? Are you looking to understand the benefits of upgrading Hadoop? If the answer is yes to any of these, this book is for you. It assumes novice-level familiarity with Hadoop.
Table of Contents (15 chapters)
14
Index

Advanced DML


The Data Manipulation Language provided by Hive is equivalent in features to any state-of-the-art SQL system. They provide standard operations such as the JOIN, GROUP BY, and UNION operations. The semantics might vary marginally depending on the operation. Different kinds of optimization hints are also present.

The GROUP BY operation

The GROUP BY operation is the same as in standard SQL, except for a few advanced features:

  • Multi-Group-By Inserts: It is possible to have multiple GROUP BY clauses within a single query. The output can be written to multiple tables or HDFS files. For example, the following query is possible:

    FROM src_table INSERT OVERWRITE TABLE id_count SELECT id, COUNT(id) GROUP BY id INSERT OVERWRITE TABLE id_sum SELECT id, SUM(id_value) GROUP BY id;
  • Map-side aggregation for GROUP BY: By setting the hive.map.aggr property to true, it is possible to enforce one level of aggregation on the Map tasks. This will yield a better-performing query.

ORDER BY versus SORT BY...