7. Performance Considerations | Apache Hive Essentials

Sign In Start Free Trial

Book Overview & Buying
Table Of Contents

Apache Hive Essentials

By : Dayong Du

4.5 (4)

Apache Hive Essentials

4.5 (4)

By: Dayong Du

Overview of this book

If you are a data analyst, developer, or simply someone who wants to use Hive to explore and analyze data in Hadoop, this is the book for you. Whether you are new to big data or an expert, with this book, you will be able to master both the basic and the advanced features of Hive. Since Hive is an SQL-like language, some previous experience with the SQL language and databases is useful to have a better understanding of this book.

Preface

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Free Chapter

1. Overview of Big Data and Hive

1. Overview of Big Data and Hive

A short history

Introducing big data

Relational and NoSQL database versus Hadoop

Batch, real-time, and stream processing

Overview of the Hadoop ecosystem

Hive overview

Summary

2. Setting Up the Hive Environment

2. Setting Up the Hive Environment

Installing Hive from Apache

Installing Hive from vendor packages

Starting Hive in the cloud

Using the Hive command line and Beeline

The Hive-integrated development environment

Summary

3. Data Definition and Description

3. Data Definition and Description

Understanding Hive data types

Data type conversions

Hive Data Definition Language

Hive database

Hive internal and external tables

Hive partitions

Hive buckets

Hive views

Summary

4. Data Selection and Scope

4. Data Selection and Scope

The SELECT statement

The INNER JOIN statement

The OUTER JOIN and CROSS JOIN statements

Special JOIN – MAPJOIN

Set operation – UNION ALL

Summary

5. Data Manipulation

5. Data Manipulation

Data exchange – LOAD

Data exchange – INSERT

Data exchange – EXPORT and IMPORT

ORDER and SORT

Operators and functions

Transactions

Summary

6. Data Aggregation and Sampling

6. Data Aggregation and Sampling

Basic aggregation – GROUP BY

Advanced aggregation – GROUPING SETS

Advanced aggregation – ROLLUP and CUBE

Aggregation condition – HAVING

Analytic functions

Sampling

Summary

7. Performance Considerations

7. Performance Considerations

Performance utilities

Design optimization

Data file optimization

Job and query optimization

Summary

8. Extensibility Considerations

8. Extensibility Considerations

User-defined functions

Streaming

SerDe

Summary

9. Security Considerations

9. Security Considerations

Authentication

Authorization

Encryption

Summary

10. Working with Other Tools

10. Working with Other Tools

JDBC / ODBC connector

HBase

Hue

HCatalog

ZooKeeper

Oozie

Hive roadmap

Summary

Index

Index

Data file optimization

Data file optimization covers the performance improvement on the data files in terms of file format, compression, and storage.

File format

Hive supports TEXTFILE, SEQUENCEFILE, RCFILE, ORC, and PARQUET file formats. The three ways to specify the file format are as follows:

CREATE TABLE ... STORE AS <File_Format>
ALTER TABLE ... [PARTITION partition_spec] SET FILEFORMAT <File_Format>
SET hive.default.fileformat=<File_Format> --default fileformat for table

Here, <File_Type> is TEXTFILE, SEQUENCEFILE, RCFILE, ORC, and PARQUET.

We can load a text file directly to a table with the TEXTFILE format. To load data to the table with other file formats, we need to load the data to a TEXTFILE format table first. Then, use INSERT OVERWRITE TABLE <target_file_format_table> SELECT * FROM <text_format_source_table> to convert and insert the data to the file format as expected.

The file formats supported by Hive and their optimizations are as follows...

CONTINUE READING

83

Tech Concepts

36

Programming languages

73

Tech Tools

Unlimited access to the largest independent learning library in tech of over 8,000 expert-authored tech books and videos.

Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.

50+ new titles added per month and exclusive early access to books as they are being written.

Apache Hive Essentials

Search

Your notes and bookmarks