Book Image

Modern Big Data Processing with Hadoop

By : V Naresh Kumar, Manoj R Patil, Prashant Shindgikar

Book Image

Modern Big Data Processing with Hadoop

By: V Naresh Kumar, Manoj R Patil, Prashant Shindgikar

Overview of this book

The complex structure of data these days requires sophisticated solutions for data transformation, to make the information more accessible to the users.This book empowers you to build such solutions with relative ease with the help of Apache Hadoop, along with a host of other Big Data tools. This book will give you a complete understanding of the data lifecycle management with Hadoop, followed by modeling of structured and unstructured data in Hadoop. It will also show you how to design real-time streaming pipelines by leveraging tools such as Apache Spark, and build efficient enterprise search solutions using Elasticsearch. You will learn to build enterprise-grade analytics solutions on Hadoop, and how to visualize your data using tools such as Apache Superset. This book also covers techniques for deploying your Big Data solutions on the cloud Apache Ambari, as well as expert techniques for managing and administering your Hadoop cluster. By the end of this book, you will have all the knowledge you need to build expert Big Data systems.

Preface

Who this book is for

What this book covers

To get the most out of this book

Free Chapter

Enterprise Data Architecture Principles

Enterprise Data Architecture Principles

Data architecture principles

The importance of metadata

Data governance

Data as a Service

Evolution data architecture with Hadoop

Hadoop Life Cycle Management

Hadoop Life Cycle Management

Hadoop Design Consideration

Hadoop Design Consideration

Understanding data structure principles

Installing Hadoop cluster

Exploring HDFS architecture

Introducing YARN

Configuring HDFS high availability

Configuration of HA NameNodes with QJM

Hadoop cluster composition

Best practices Hadoop deployment

Hadoop file formats

Data Movement Techniques

Data Movement Techniques

Batch processing versus real-time processing

Data Modeling in Hadoop

Data Modeling in Hadoop

Supported datatypes

Hive architecture

Hive data model management

JSON documents using Hive

Designing Real-Time Streaming Data Pipelines

Designing Real-Time Streaming Data Pipelines

Real-time streaming concepts

Real-time streaming components

Other popular real-time data streaming frameworks

Apache Flink versus Spark

Apache Spark versus Storm

Large-Scale Data Processing Frameworks

Large-Scale Data Processing Frameworks

Hadoop MapReduce

Building Enterprise Search Platform

Building Enterprise Search Platform

The data search concept

The need for an enterprise search engine

How to index documents in Elasticsearch?

Elasticsearch-supported data types

Designing Data Visualization Solutions

Designing Data Visualization Solutions

Data visualization

Practical data visualization in Hadoop

Developing Applications Using the Cloud

Developing Applications Using the Cloud

What is the Cloud?

Available technologies in the Cloud

Planning the Cloud infrastructure

Building a Hadoop cluster in the Cloud

Data access in the Cloud

Production Hadoop Cluster Deployment

Production Hadoop Cluster Deployment

Apache Ambari architecture

Setting up a Hadoop cluster with Ambari

Hadoop clusters

Customer Reviews

5 star

0

4 star

0

3 star

0

2 star

0

1 star

0

Best practices Hadoop deployment

Following are some best practices to be followed for Hadoop deployment:

Start small: Like other software projects, an implementation Hadoop also involves risks and cost. It's always better to set up a small Hadoop cluster of four nodes. This small cluster can be set up as proof of concept (POC). Before using any Hadoop component, it can be added to the existing Hadoop POC cluster as proof of technology (POT). It allows the infrastructure and development team to understand big data project requirements. After successful completion of POC and POT, additional nodes can be added to the existing cluster.
Hadoop cluster monitoring: Proper monitoring of the NameNode and all DataNodes is required to understand the health of the cluster. It helps to take corrective actions in the event of node problems. If a service goes down, timely action can help...