Book Image

Apache Hadoop 3 Quick Start Guide

By : Hrishikesh Vijay Karambelkar
Book Image

Apache Hadoop 3 Quick Start Guide

By: Hrishikesh Vijay Karambelkar

Overview of this book

Apache Hadoop is a widely used distributed data platform. It enables large datasets to be efficiently processed instead of using one large computer to store and process the data. This book will get you started with the Hadoop ecosystem, and introduce you to the main technical topics, including MapReduce, YARN, and HDFS. The book begins with an overview of big data and Apache Hadoop. Then, you will set up a pseudo Hadoop development environment and a multi-node enterprise Hadoop cluster. You will see how the parallel programming paradigm, such as MapReduce, can solve many complex data processing problems. The book also covers the important aspects of the big data software development lifecycle, including quality assurance and control, performance, administration, and monitoring. You will then learn about the Hadoop ecosystem, and tools such as Kafka, Sqoop, Flume, Pig, Hive, and HBase. Finally, you will look at advanced topics, including real time streaming using Apache Storm, and data analytics using Apache Spark. By the end of the book, you will be well versed with different configurations of the Hadoop 3 cluster.

Related Content you might be interested in

Current Title:

Small book image
Apache Hadoop 3 Quick Start Guide
Hrishikesh Vijay Karambelkar
Oct, 2018 | 220 pages
Small book image
Mastering Hadoop 3
Manish Kumar | Chanchal Singh
Feb, 2019 | 544 pages
Small book image
Hadoop 2.x Administration Cookbook
Gurmukh Singh
May, 2017 | 348 pages
Small book image
HBase High Performance Cookbook
Ruchir Choudhry
Jan, 2017 | 350 pages
Table of Contents (10 chapters)
Preface
Preface
Who this book is for
What this book covers
To get the most out of this book
Get in touch
Free Chapter
1
Hadoop 3.0 - Background and Introduction
Hadoop 3.0 - Background and Introduction
How it all started
What Hadoop is and why it is important
How Apache Hadoop works
Hadoop 3.0 releases and new features
Choosing the right Hadoop distribution
Summary
2
Planning and Setting Up Hadoop Clusters
Planning and Setting Up Hadoop Clusters
Technical requirements
Prerequisites for Hadoop setup
Running Hadoop in standalone mode
Setting up a pseudo Hadoop cluster
Planning and sizing clusters
Setting up Hadoop in cluster mode
Diagnosing the Hadoop cluster
Summary
3
Deep Dive into the Hadoop Distributed File System
Deep Dive into the Hadoop Distributed File System
Technical requirements
How HDFS works
Key features of HDFS
Data flow patterns of HDFS
HDFS configuration files
Hadoop filesystem CLIs
Working with data structures in HDFS
Summary
4
Developing MapReduce Applications
Developing MapReduce Applications
Technical requirements
How MapReduce works
Configuring a MapReduce environment
Understanding Hadoop APIs and packages
Setting up a MapReduce project
Deep diving into MapReduce APIs
Compiling and running MapReduce jobs
Streaming in MapReduce programming
Summary
5
Building Rich YARN Applications
Building Rich YARN Applications
Technical requirements
Understanding YARN architecture
Key features of YARN
Configuring the YARN environment in a cluster
Working with YARN distributed CLI
Deep dive with YARN application framework
Building and monitoring a YARN application on a cluster
Summary
6
Monitoring and Administration of a Hadoop Cluster
Monitoring and Administration of a Hadoop Cluster
Roles and responsibilities of Hadoop administrators
Planning your distributed cluster
Resource management in Hadoop
High availability of Hadoop
Securing Hadoop clusters
Performing routine tasks
Summary
7
Demystifying Hadoop Ecosystem Components
Demystifying Hadoop Ecosystem Components
Technical requirements
Understanding Hadoop's Ecosystem
Working with Apache Kafka
Writing Apache Pig scripts
Transferring data with Sqoop
Writing Flume jobs
Understanding Hive
Using HBase for NoSQL storage
Summary
8
Advanced Topics in Apache Hadoop
Advanced Topics in Apache Hadoop
Technical requirements
Hadoop use cases in industries
Advanced Hadoop data storage file formats
Real-time streaming with Apache Storm
Data analytics with Apache Spark
Summary
9
Other Books You May Enjoy
Other Books You May Enjoy
Leave a review - let other readers know what you think

Setting up a MapReduce project

In this section, we will learn how to create the environment to start writing applications for MapReduce programming. The programming is typically done in Java. The development of a MapReduce application follows standard Java development principles as follows:

  1. Usually, developers write the programs in a development environment such as Eclipse or NetBeans.
  2. Developers do unit testing usually with a small subset of data. In case of failure, they can run an IDE Debugger to do fault identification.
  3. It is then packaged in JAR files and is tested in a standalone fashion for functionality.
  4. Developers should ideally write unit test cases to test each functionality.
  5. Once it is tested in standalone mode, developers should test it in a cluster or pseudo-distributed environment with full datasets. This will expose more problems, and they can be fixed. Here debugging...
Previous Section
End of Section 6
Next Section