Book Image

Hadoop Essentials

By : Shiva Achari
Book Image

Hadoop Essentials

By: Shiva Achari

Overview of this book

This book jumps into the world of Hadoop and its tools, to help you learn how to use them effectively to optimize and improve the way you handle Big Data. Starting with the fundamentals Hadoop YARN, MapReduce, HDFS, and other vital elements in the Hadoop ecosystem, you will soon learn many exciting topics such as MapReduce patterns, data management, and real-time data analysis using Hadoop. You will also explore a number of the leading data processing tools including Hive and Pig, and learn how to use Sqoop and Flume, two of the most powerful technologies used for data ingestion. With further guidance on data streaming and real-time analytics with Storm and Spark, Hadoop Essentials is a reliable and relevant resource for anyone who understands the difficulties - and opportunities - presented by Big Data today. With this guide, you'll develop your confidence with Hadoop, and be able to use the knowledge and skills you learn to successfully harness its unparalleled capabilities.

Related Content you might be interested in

Current Title:

Small book image
Hadoop Essentials
Shiva Achari
Apr, 2015 | 194 pages
No titles found
Table of Contents (15 chapters)
Hadoop Essentials
Hadoop Essentials
Credits
Credits
About the Author
About the Author
Acknowledgments
Acknowledgments
About the Reviewers
About the Reviewers
www.PacktPub.com
www.PacktPub.com
Preface
Preface
Free Chapter
1
Introduction to Big Data and Hadoop
Introduction to Big Data and Hadoop
V's of big data
Understanding big data
Who is creating big data?
Big data use case patterns
Hadoop
Pillars of Hadoop
Data access components
Data storage component
Data ingestion in Hadoop
Streaming and real-time analysis
Summary
2
Hadoop Ecosystem
Hadoop Ecosystem
Traditional systems
The Hadoop use cases
Hadoop's basic data flow
Hadoop integration
The Hadoop ecosystem
Distributed filesystem
Distributed programming
NoSQL databases
Data ingestion
Service programming
Scheduling
Data analytics and machine learning
System management
Summary
3
Pillars of Hadoop – HDFS, MapReduce, and YARN
Pillars of Hadoop – HDFS, MapReduce, and YARN
HDFS
MapReduce
YARN
Summary
4
Data Access Components – Hive and Pig
Data Access Components – Hive and Pig
Need of a data processing tool on Hadoop
Pig
Hive
Summary
5
Storage Component – HBase
Storage Component – HBase
An Overview of HBase
Advantages of HBase
The Architecture of HBase
The HBase data model
The Schema design
The Write pipeline
The Read pipeline
Compaction
Splitting
Commands
HBase Hive integration
Performance tuning
Summary
6
Data Ingestion in Hadoop – Sqoop and Flume
Data Ingestion in Hadoop – Sqoop and Flume
Data sources
Challenges in data ingestion
Sqoop
Connectors and drivers
Sqoop 1 architecture
Sqoop 2 architecture
Imports
Exports
Apache Flume
Flume architecture
Examples of configuring Flume
Summary
7
Streaming and Real-time Analysis – Storm and Spark
Streaming and Real-time Analysis – Storm and Spark
An introduction to Storm
Storm topology
Storm on YARN
Topology configuration example
An introduction to Spark
Spark framework
Spark architecture
Operations in Spark
Spark example
Summary
Index
Index

HBase Hive integration

Analysts usually prefer a Hive environment due to the comfort of SQL-like syntax. HBase is well integrated with Hive, using the StorageHandler that Hive interfaces with. The create table syntax in Hive will look like the following:

CREATE EXTERNAL TABLE hbase_table_1(key int, value string) 
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,ColumnFamily:Column1, columnFalimy:column2")
TBLPROPERTIES ("hbase.table.name" = "xyz");

Let us understand the syntax and the keywords used for the table:

  • EXTERNAL: This is used if the table in HBase exists already, or if the table in HBase is new and you want Hive to manage only the metadata and not the actual data.

  • STORED BY: The HBaseStorageHandler has to be used to handle the input and output from HBase.

  • SERDEPROPERTIES: Hive column to HBase ColumnFamily:Column mapping has to be specified here. In this example, key maps as a rowkey and value maps to val column of...

Previous Section
End of Section 12
Next Section