4. Advanced Hive | Mastering Hadoop

Sign In Start Free Trial

Book Overview & Buying
Table Of Contents

Mastering Hadoop

By : Karanth

4 (3)

Mastering Hadoop

4 (3)

By: Karanth

Overview of this book

Do you want to broaden your Hadoop skill set and take your knowledge to the next level? Do you wish to enhance your knowledge of Hadoop to solve challenging data processing problems? Are your Hadoop jobs, Pig scripts, or Hive queries not working as fast as you intend? Are you looking to understand the benefits of upgrading Hadoop? If the answer is yes to any of these, this book is for you. It assumes novice-level familiarity with Hadoop.

Preface

Preface

What this book covers

What you need for this book?

Who this book is for

Conventions

Reader feedback

Customer support

Free Chapter

1. Hadoop 2.X

1. Hadoop 2.X

The inception of Hadoop

The evolution of Hadoop

Hadoop 2.X

Hadoop distributions

Summary

2. Advanced MapReduce

2. Advanced MapReduce

MapReduce input

The RecordReader class

Hadoop's "small files" problem

Filtering inputs

The Map task

The Reduce task

MapReduce output

MapReduce job counters

Handling data joins

Summary

3. Advanced Pig

3. Advanced Pig

Pig versus SQL

Different modes of execution

Complex data types in Pig

Compiling Pig scripts

Development and debugging aids

The advanced Pig operators

User-defined functions

Pig performance optimizations

Best practices

Summary

4. Advanced Hive

4. Advanced Hive

The Hive architecture

Data types

File formats

The data model

Hive query optimizers

Advanced DML

UDF, UDAF, and UDTF

Summary

5. Serialization and Hadoop I/O

5. Serialization and Hadoop I/O

Data serialization in Hadoop

Avro serialization

File formats

Compression

Summary

6. YARN – Bringing Other Paradigms to Hadoop

6. YARN – Bringing Other Paradigms to Hadoop

The YARN architecture

Developing YARN applications

Monitoring YARN

Job scheduling in YARN

YARN commands

Summary

7. Storm on YARN – Low Latency Processing in Hadoop

7. Storm on YARN – Low Latency Processing in Hadoop

Batch processing versus streaming

Apache Storm

Storm on YARN

Summary

8. Hadoop on the Cloud

8. Hadoop on the Cloud

Cloud computing characteristics

Hadoop on the cloud

Amazon Elastic MapReduce (EMR)

Summary

9. HDFS Replacements

9. HDFS Replacements

HDFS – advantages and drawbacks

Amazon AWS S3

Implementing a filesystem in Hadoop

Implementing an S3 native filesystem in Hadoop

Summary

10. HDFS Federation

10. HDFS Federation

Limitations of the older HDFS architecture

Architecture of HDFS Federation

HDFS high availability

HDFS block placement

Summary

11. Hadoop Security

11. Hadoop Security

The security pillars

Authentication in Hadoop

Authorization in Hadoop

Data confidentiality in Hadoop

Audit logging in Hadoop

Summary

12. Analytics Using Hadoop

12. Analytics Using Hadoop

Data analytics workflow

Machine learning

Apache Mahout

Document analysis using Hadoop and Mahout

RHadoop

Summary

A. Hadoop for Microsoft Windows

A. Hadoop for Microsoft Windows

Deploying Hadoop on Microsoft Windows

Summary

Index

Index

Data types

Hive supports all the primitive numeric data types such as TINYINT, SMALLINT, INT, BIGINT, FLOAT, DOUBLE, and DECIMAL. In addition to these primitive data types, Hive also supports string types such as CHAR, VARCHAR, and STRING data types. Like SQL, the time indicator data types such as TIMESTAMP and DATE are present. BOOLEAN and BINARY miscellaneous types are available too.

A number of complex types are also available. Complex types can be composed from other primitive or complex types. The complex types available are as follows:

STRUCTS: These are groupings of data elements similar to a C-struct. The dot notation is used to dereference elements within a struct. A field within column C defined as STRUCT {x INT, y STRING} can be accessed as A.x or A.y.
The syntax for this is STRUCT<field_name : data_type>
MAPS: These are key-value data types; providing the key within square braces can access a value. A value of a map column M that maps from key x to value y can be accessed...

CONTINUE READING

83

Tech Concepts

36

Programming languages

73

Tech Tools

Unlimited access to the largest independent learning library in tech of over 8,000 expert-authored tech books and videos.

Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.

50+ new titles added per month and exclusive early access to books as they are being written.

Mastering Hadoop

Search

Your notes and bookmarks