Book Image

Data Lake Development with Big Data

By : Pradeep Pasupuleti, Beulah Salome Purra
Book Image

Data Lake Development with Big Data

By: Pradeep Pasupuleti, Beulah Salome Purra

Overview of this book

A Data Lake is a highly scalable platform for storing huge volumes of multistructured data from disparate sources with centralized data management services. This book explores the potential of Data Lakes and explores architectural approaches to building data lakes that ingest, index, manage, and analyze massive amounts of data using batch and real-time processing frameworks. It guides you on how to go about building a Data Lake that is managed by Hadoop and accessed as required by other Big Data applications. This book will guide readers (using best practices) in developing Data Lake's capabilities. It will focus on architect data governance, security, data quality, data lineage tracking, metadata management, and semantic data tagging. By the end of this book, you will have a good understanding of building a Data Lake for Big Data.
Table of Contents (13 chapters)

Chapter 5. Data Governance

In the preceding chapter, you understood the various aspects of Data Consumption in detail, such as Data Discovery and Data Provisioning. We also understood architectural guidance on choosing the Big Data tools and technologies that can be used for Data Discovery and Data Provisioning.

In this chapter, you will understand the details of Data Governance; the following topics will be covered:

  • Learn how to deal with management, usability, security, integrity, and the availability of the data in Data Lake

  • Dive deep into the various Data Governance disciplines such as metadata management, lineage tracking, and data lifecycle management that are commonly applied on the data as it flows through each tier of Data Lake

  • Explore how the current Data Lake could evolve in a futuristic setting

The following figure represents the end-state architecture of the Data Lake as discussed in Chapter 1, The Need for Data Lake. As shown in the following figure, we will discuss the highlighted...