Book Image

Data Lake Development with Big Data

Book Image

Data Lake Development with Big Data

Overview of this book

A Data Lake is a highly scalable platform for storing huge volumes of multistructured data from disparate sources with centralized data management services. This book explores the potential of Data Lakes and explores architectural approaches to building data lakes that ingest, index, manage, and analyze massive amounts of data using batch and real-time processing frameworks. It guides you on how to go about building a Data Lake that is managed by Hadoop and accessed as required by other Big Data applications. This book will guide readers (using best practices) in developing Data Lake's capabilities. It will focus on architect data governance, security, data quality, data lineage tracking, metadata management, and semantic data tagging. By the end of this book, you will have a good understanding of building a Data Lake for Big Data.
Table of Contents (13 chapters)

Summary


This chapter explained in detail Data Governance and the ways to manage data with focus on its availability, usability, integrity, retention, and security. We started with understanding data governance and why it is needed and then understood how data governance on the Data Lake is far more efficient when compared to traditional governance. We also took a look at a few practical scenarios to comprehend the real-life use cases of Data Governance.

We took a deep dive into Data Governance and its components, such as data security and privacy, metadata management and lineage tracking, Information Lifecycle Management, and how they cut across all the three tiers of Data Lake, such as Data Intake, management, and consumption. In the subsequent sections, we took a look at the various Big Data tools and technologies that can be used to perform data governance to help you in decision making and to arrive at the set of technologies that can be used for specific use cases by giving an overview...