We live in a connected digital era and we are witnessing unprecedented growth of data. Organizations that are able to analyze big data are demonstrating significant return on investment by detecting fraud, improved operations, and reduced time to analyze with a scale-out architecture such as Hadoop. Azure HDInsight is an enterprise-ready distribution of Hadoop hosted in the cloud and provides advanced integration with Excel and .NET without the need to buy or maintain physical hardware.
This book is your guide to building a modern data architecture using HDInsight to enable your organization to gain insights from various sources, including smart-connected devices, databases, and social media. This book will take you through a journey of building the next generation Enterprise Data Lake that consists of ingestion, transformation, and analysis of big data with a specific use case that can apply to almost any organization.
This book has working code that developers can leverage and extend in order to fit their use cases with additional references for self-learning.
Chapter 1, Hadoop and HDInsight in a Heartbeat, covers the business value and the reason behind the big data hype. It provides a primer on Apache Hadoop, core concepts with HDFS, YARN, and the Hadoop 2.x ecosystem. Next, it discusses the Microsoft HDInsight platform, its key benefits, and deployment options.
Chapter 2, Enterprise Data Lake using HDInsight, covers the main points of the current Enterprise Data Warehouse and provides a path for an enterprise Data Lake based on the Hadoop platform. Additionally, it explains a use case built on the Azure HDInsight service.
Chapter 3, HDInsight Service on Azure, walks you through the steps for provisioning Azure HDInsight. Next, it explains how to explore, monitor, and delete the cluster using the Azure management portal. Next, it provides tools for developers to verify the cluster using a sample program and develop it using HDInsight Emulator.
Chapter 4, Administering Your HDInsight Cluster, covers steps to administer the HDInsight cluster using remote desktop connection to the head node of the cluster. It includes management of Azure Blob storage and introduces you to the Azure scripting environment known as Azure PowerShell.
Chapter 5, Ingest and Organize Data Lake, introduces you to an end-to-end Data Lake solution with a near real life size project and then focuses on various options to ingest data to a HDInsight cluster, including HDFS commands, Azure PowerShell, CloudExplorer, and Sqoop. Next, it provides details on how to organize data using Apache HCatalog. This chapter uses a real life size sample airline project to explain the various concepts.
Chapter 6, Transform Data in the Data Lake, provides you with various options to transform data, including MapReduce, Hive, and Pig. Additionally, it discusses Oozie and Spark, which are also commonly used for transformation. Throughout the chapter, you will be guided with a detailed code for the sample airline project.
Chapter 7, Analyze and Report from Data Lake, provides you with details on how to access and analyze data from the sample airline project using Excel Hive ODBC driver, Excel Power Query, Powerpivot, and PowerMap. Additionally, it discusses RHadoop, Giraph, and Mahout as alternatives to analyze data in the cluster.
Chapter 8, HDInsight 3.1 New Features, provides you with new features that are added to the evolving HDInsight platform with sample use cases for HBase, Tez, and Storm.
Chapter 9, Strategy for a Successful Data Lake Implementation, covers the key challenges for building a production Data Lake and provides guidance on the success path for a sustainable Data Lake. This chapter provides recommendations on architecture, organization, and links to online resources.
For this book, the following are the prerequisites:
To build an HDInsight cluster using the Azure cloud service, you will need an Azure account and a laptop with Windows Remote Desktop software to connect to the cluster
For Excel-based exercises, you will need Office 2013/Excel 2013/Office 365 ProPlus/Office 2010 Professional Plus
For HDInsight Emulator, which is suited for local development, you will need a Windows laptop with one of these operating systems: Windows 7 Service Pack 1/Windows Server 2008 R2 Service Pack 1/Windows 8/Windows Server 2012.
This book is designed for data architects, developers, managers, and business users who want to modernize their data architectures leveraging the HDInsight distribution of Hadoop. It guides you through the business values of big data, the main points of current EDW (Enterprise Data Warehouse), steps for building the next generation Data Lake, and development tools with real life examples.
The book explains the journey to a Data Lake with a modular approach for ingesting, transforming, and reporting on a Data Lake leveraging HDInsight platform and Excel for powerful analysis and reporting.
In this book, you will find a number of text styles that distinguish between different kinds of information. Here are some examples of these styles and an explanation of their meaning.
Code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles are shown as follows: "I have selected hdind and the complete URL is hdind.azurehdinsight.net."
Any command-line input or output is written as follows:
# Import PublishSettingsFile that was saved from last step Import-AzurePublishSettingsFile "C:\Users\Administrator\Downloads\Pay-As-You-Go-Free Trial-11-21-2014-credentials.publishsettings"
New terms and important words are shown in bold. Words that you see on the screen, for example, in menus or dialog boxes, appear in the text like this: "You can select the desired configuration Two Head Nodes on an Extra Large (A4) instance included or Two Head Nodes on a Large (A3) instance included."
Feedback from our readers is always welcome. Let us know what you think about this book—what you liked or disliked. Reader feedback is important for us as it helps us develop titles that you will really get the most out of.
To send us general feedback, simply e-mail <[email protected]>, and mention the book's title in the subject of your message.
If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, see our author guide at www.packtpub.com/authors.
Now that you are the proud owner of a Packt book, we have a number of things to help you to get the most from your purchase.
You can download the example code files from your account at http://www.packtpub.com for all the Packt Publishing books you have purchased. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you.
Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you find a mistake in one of our books—maybe a mistake in the text or the code—we would be grateful if you could report this to us. By doing so, you can save other readers from frustration and help us improve subsequent versions of this book. If you find any errata, please report them by visiting http://www.packtpub.com/submit-errata, selecting your book, clicking on the Errata Submission Form link, and entering the details of your errata.
Once your errata are verified, your submission will be accepted and the errata will be uploaded to our website or added to any list of existing errata under the Errata section of that title.
To view the previously submitted errata, go to https://www.packtpub.com/books/content/support and enter the name of the book in the search field. The required information will appear under the Errata section.
Piracy of copyrighted material on the Internet is an ongoing problem across all media. At Packt, we take the protection of our copyright and licenses very seriously. If you come across any illegal copies of our works in any form on the Internet, please provide us with the location address or website name immediately so that we can pursue a remedy.
Please contact us at <[email protected]> with a link to the suspected pirated material.
We appreciate your help in protecting our authors and our ability to bring you valuable content.
If you have a problem with any aspect of this book, you can contact us at <[email protected]>, and we will do our best to address the problem.