Book Image

Data Lake for Enterprises

By : Vivek Mishra, Tomcy John, Pankaj Misra
Book Image

Data Lake for Enterprises

By: Vivek Mishra, Tomcy John, Pankaj Misra

Overview of this book

The term "Data Lake" has recently emerged as a prominent term in the big data industry. Data scientists can make use of it in deriving meaningful insights that can be used by businesses to redefine or transform the way they operate. Lambda architecture is also emerging as one of the very eminent patterns in the big data landscape, as it not only helps to derive useful information from historical data but also correlates real-time data to enable business to take critical decisions. This book tries to bring these two important aspects — data lake and lambda architecture—together. This book is divided into three main sections. The first introduces you to the concept of data lakes, the importance of data lakes in enterprises, and getting you up-to-speed with the Lambda architecture. The second section delves into the principal components of building a data lake using the Lambda architecture. It introduces you to popular big data technologies such as Apache Hadoop, Spark, Sqoop, Flume, and ElasticSearch. The third section is a highly practical demonstration of putting it all together, and shows you how an enterprise data lake can be implemented, along with several real-world use-cases. It also shows you how other peripheral components can be added to the lake to make it more efficient. By the end of this book, you will be able to choose the right big data technologies using the lambda architectural patterns to build your enterprise data lake.

Related Content you might be interested in

Current Title:

Small book image
Data Lake for Enterprises
Vivek Mishra | Tomcy John | Pankaj Misra
May, 2017 | 596 pages
Small book image
Modern Big Data Processing with Hadoop
V Naresh Kumar | Prashant Shindgikar
Mar, 2018 | 394 pages
Small book image
Mastering Hadoop 3
Manish Kumar | Chanchal Singh
Feb, 2019 | 544 pages
Small book image
Apache Hadoop 3 Quick Start Guide
Hrishikesh Vijay Karambelkar
Oct, 2018 | 220 pages
Table of Contents (23 chapters)
Title Page
Title Page
Credits
Credits
Foreword
Foreword
About the Authors
About the Authors
About the Reviewers
About the Reviewers
www.PacktPub.com
www.PacktPub.com
Customer Feedback
Customer Feedback
Preface
Preface
Part 1 - Overview
Part 1 - Overview
Part 2 - Technical Building blocks of Data Lake
Part 2 - Technical Building blocks of Data Lake
Part 3 - Bringing It All Together
Part 3 - Bringing It All Together
Free Chapter
1
Introduction to Data
Introduction to Data
Exploring data
What is Enterprise Data?
Enterprise Data Management
Big data concepts
Relevance of data
Quality of data
Where does this data live in an enterprise?
Enterprise’s current state
Enterprise digital transformation
Data lake use case enlightenment
Summary
2
Comprehensive Concepts of a Data Lake
Comprehensive Concepts of a Data Lake
What is a Data Lake?
How does a Data Lake help enterprises?
How Data Lake works?
Differences between Data Lake and Data Warehouse
Approaches to building a Data Lake
Lambda Architecture-driven Data Lake
Summary
3
Lambda Architecture as a Pattern for Data Lake
Lambda Architecture as a Pattern for Data Lake
What is Lambda Architecture?
History of Lambda Architecture
Principles of Lambda Architecture
Components of a Lambda Architecture
Complete working of a Lambda Architecture
Advantages of Lambda Architecture
Disadvantages of Lambda Architectures
Technology overview for Lambda Architecture
Applied lambda
Working examples of Lambda Architecture
Kappa architecture
Summary
4
Applied Lambda for Data Lake
Applied Lambda for Data Lake
Knowing Hadoop distributions
Selection factors for a big data stack for enterprises
Batch layer for data processing
Serving layer
Summary
5
Data Acquisition of Batch Data using Apache Sqoop
Data Acquisition of Batch Data using Apache Sqoop
Context in data lake - data acquisition
Why Apache Sqoop
Workings of Sqoop
Sqoop connectors
Sqoop support for HDFS
Sqoop working example
When to use Sqoop
When not to use Sqoop
Real-time Sqooping: a possibility?
Other options
Summary
6
Data Acquisition of Stream Data using Apache Flume
Data Acquisition of Stream Data using Apache Flume
Context in Data Lake: data acquisition
Why Flume?
Flume architecture principles
The Flume Architecture
Flume event - Stream Data
Flume agent
Flume source
Flume Channel
Flume sink
Flume configuration
Flume transaction management
Other flume components
Context Routing
Flume working example
When to use Flume
When not to use Flume
Other options
Summary
7
Messaging Layer using Apache Kafka
Messaging Layer using Apache Kafka
Context in Data Lake - messaging layer
Why Apache Kafka
Kafka architecture
Other Kafka components
Kafka programming interface
Producer and consumer reliability
Kafka security
Kafka as message-oriented middleware
Scale-out architecture with Kafka
Kafka connect
Kafka working example
When to use Kafka
When not to use Kafka
Other options
Summary
8
Data Processing using Apache Flink
Data Processing using Apache Flink
Context in a Data Lake - Data Ingestion Layer
Why Apache Flink?
Working of Flink
Flink API’s
Flink working example
When to use Flink
When not to use Flink
Other options
Summary
9
Data Store Using Apache Hadoop
Data Store Using Apache Hadoop
Context for Data Lake - Data Storage and lambda Batch layer
Why Hadoop?
Working of Hadoop
Hadoop ecosystem
Hadoop distributions
HDFS and formats
Hadoop for near real-time applications
Hadoop deployment modes
Hadoop working examples
When not to use Hadoop
Other Hadoop Processing Options
Summary
10
Indexed Data Store using Elasticsearch
Indexed Data Store using Elasticsearch
Context in Data Lake: data storage and lambda speed layer
What is Elasticsearch?
Why Elasticsearch
Working of Elasticsearch
Elastic Stack
Elastic Cloud
Elasticsearch DSL (Query DSL)
Nodes in Elasticsearch
Elasticsearch and relational database
Elasticsearch ecosystem
Elasticsearch deployment options
Clients for Elasticsearch
Elasticsearch for fast streaming layer
Elasticsearch as a data source
Elasticsearch for content indexing
Elasticsearch and Hadoop
Elasticsearch working example
Indexing Documents
Getting Indexed Document
Searching Documents
Updating Documents
Deleting a document
Elasticsearch in purview of SCV use case
11
Data Lake Components Working Together
Data Lake Components Working Together
Where we stand with Data Lake
Core architecture principles of Data Lake
Challenges faced by enterprise Data Lake
Expectations from Data Lake
Data Lake for other activities
Knowing more about data storage
Knowing more about Data processing
Thoughts on data security
Thoughts on data encryption
Metadata management and governance
Thoughts on Data Auditing
Thoughts on data traceability
Knowing more about Serving Layer
Summary
12
Data Lake Use Case Suggestions
Data Lake Use Case Suggestions
Establishing cybersecurity practices in an enterprise
Know the customers dealing with your enterprise
Bring efficiency in warehouse management 
Developing a brand and marketing of the enterprise
Achieve a higher degree of personalization with customers
Bringing IoT data analysis at your fingertips
More practical and useful data archival
Compliment the existing data warehouse infrastructure
Achieving telecom security and regulatory compliance
Summary

Part 3 - Bringing It All Together

This part of the books brings together technical components from part one and two of this book to give you a holistic picture of Data Lakes. We will bring in additional concepts and technologies in a brief fashion, so that you can explore those aspects in more detail according to your enterprise requirements. Again, delving deep into the technologies covered in this chapter is out of the scope of this book. But we want you to be aware of these additional technologies and how they can be brought into our Data Lake implementation if the need arises. It consists of two chapters, and each chapter has a goal well defined to be achieved, as detailed below.

Chapter 11, Data Lake Components Working Together, after introducing reader into Data Lake, Lambda Architecture, various technologies, this chapter brings the whole puzzle together and brings in a holistic picture to the reader. The reader at this stage should feel accomplished and can take in the codebase as is into the organization and show it working. In this chapter, the reader, would realize how to integrate various aspects of Data Lake to implement a fully functional Data Lake. The reader will also realize the completeness of Data Lake with working examples that would combine all the learnings from previous chapters into a running implementation.

Chapter 12, Data Lake Use Case Suggestions, throughout the book the reader is taken through a use case in the form of “Single Customer View”; however while going through the book, there are other use cases in pipeline relevant to their organization which reader can start thinking. This provoking of thought deepens into bit more during this chapter. The reader will understand and realize various use cases that can reap great benefits from a Data Lake and help optimize their cost of ownership, operations, reactiveness and help these uses with required intelligence derived from data. The reader, in this chapter, will also realize the variety of these use cases and the extents to which an enterprise Data Lake can be helpful for each of these use cases.

Previous Section
Next Chapter