Book Image

HDInsight Essentials - Second Edition

By : Rajesh Nadipalli
Book Image

HDInsight Essentials - Second Edition

By: Rajesh Nadipalli

Overview of this book

Related Content you might be interested in

Current Title:

Small book image
HDInsight Essentials - Second Edition
Rajesh Nadipalli
Jan, 2015 | 178 pages
No titles found
Table of Contents (16 chapters)
HDInsight Essentials Second Edition
HDInsight Essentials Second Edition
Credits
Credits
About the Author
About the Author
About the Reviewers
About the Reviewers
www.PacktPub.com
www.PacktPub.com
Preface
Preface
Free Chapter
1
Hadoop and HDInsight in a Heartbeat
Hadoop and HDInsight in a Heartbeat
Data is everywhere
Hadoop concepts
Hadoop distributions
HDInsight overview
Hadoop on Windows deployment options
Summary
2
Enterprise Data Lake using HDInsight
Enterprise Data Lake using HDInsight
Enterprise Data Warehouse architecture
The next generation Hadoop-based Enterprise data architecture
Journey to your Data Lake dream
Tools and technology for Hadoop ecosystem
Use case powered by Microsoft HDInsight
Summary
3
HDInsight Service on Azure
HDInsight Service on Azure
Registering for an Azure account
Azure storage
Provisioning an HDInsight cluster
HDInsight management dashboard
Exploring clusters using the remote desktop
Deleting the cluster
HDInsight Emulator for the development
Summary
4
Administering Your HDInsight Cluster
Administering Your HDInsight Cluster
Monitoring cluster health
Name Node status
Hadoop Service Availability
YARN Application Status
Azure storage management
Azure PowerShell
Summary
5
Ingest and Organize Data Lake
Ingest and Organize Data Lake
End-to-end Data Lake solution
Ingesting to Data Lake using HDFS command
Loading data to Azure Blob storage using Azure PowerShell
Loading files to Data Lake using GUI tools
Using Sqoop to move data from RDBMS to Data Lake
Organizing your Data Lake in HDFS
Managing file metadata using HCatalog
Summary
6
Transform Data in the Data Lake
Transform Data in the Data Lake
Transformation overview
Tools for transforming data in Data Lake
Transformation for the OTP project
Other tools used for transformation
Summary
7
Analyze and Report from Data Lake
Analyze and Report from Data Lake
Data access overview
Analysis using Excel and Microsoft Hive ODBC driver
Analysis using Excel Power Query
Other BI features in Excel
Ad hoc analysis using Hive
Other alternatives for analysis
Summary
8
HDInsight 3.1 New Features
HDInsight 3.1 New Features
HBase
Storm
Apache Tez
Summary
9
Strategy for a Successful Data Lake Implementation
Strategy for a Successful Data Lake Implementation
Challenges on building a production Data Lake
The success path for a production Data Lake
Architectural considerations
Online resources
Summary
Index
Index

Chapter 6. Transform Data in the Data Lake

In the previous chapter, we ingested the source data into the Data Lake. To make sense of the vast amount of raw data, a transformation procedure is required to convert it into information that can further be used by decision makers. In this chapter, we will discuss how to transform data.

The topics covered in this chapter are as follows:

  • Transformation overview

  • Tools for transforming data in a Data Lake, such as HCatalog, Hive, Pig, and MapReduce

  • Transformation of the airline on-time performance (OTP) raw data into an aggregate

  • Review results of transformation

Previous Section
End of Section 1
Next Section