Browse Library
Sign In Start Free Trial
Close icon Search icon
Account
Browse Library Advanced Search Sign In Start Free Trial

Add to playlist

Create a Playlist

Modal Close icon
You need to login to use this feature.
  • Book Overview & Buying Hadoop MapReduce v2 Cookbook - Second Edition: RAW
  • Table Of Contents Toc
  • Feedback & Rating feedback
Hadoop MapReduce v2 Cookbook - Second Edition: RAW

Hadoop MapReduce v2 Cookbook - Second Edition: RAW - Second Edition

4.4 (7)
Buy this Book
close
close
Hadoop MapReduce v2 Cookbook - Second Edition: RAW

Hadoop MapReduce v2 Cookbook - Second Edition: RAW

4.4 (7)
Buy this Book

Overview of this book

If you are a Big Data enthusiast and wish to use Hadoop v2 to solve your problems, then this book is for you. This book is for Java programmers with little to moderate knowledge of Hadoop MapReduce. This is also a one-stop reference for developers and system admins who want to quickly get up to speed with using Hadoop v2. It would be helpful to have a basic knowledge of software development using Java and a basic working knowledge of Linux.
Table of Contents (12 chapters)
close
close
close
Preface
Icon
Preface
Icon
What this book covers
Icon
What you need for this book
Icon
Who this book is for
Icon
Conventions
Icon
Reader feedback
Icon
Customer support
Lock Free Chapter
1
1. Getting Started with Hadoop v2
chevron up
Icon
1. Getting Started with Hadoop v2
Icon
Introduction
Icon
Setting up Hadoop v2 on your local machine
Icon
Writing a WordCount MapReduce application, bundling it, and running it using the Hadoop local mode
Icon
Adding a combiner step to the WordCount MapReduce program
Icon
Setting up HDFS
Icon
Setting up Hadoop YARN in a distributed cluster environment using Hadoop v2
Icon
Setting up Hadoop ecosystem in a distributed cluster environment using a Hadoop distribution
Icon
HDFS command-line file operations
Icon
Running the WordCount program in a distributed cluster environment
Icon
Benchmarking HDFS using DFSIO
Icon
Benchmarking Hadoop MapReduce using TeraSort
2
2. Cloud Deployments – Using Hadoop YARN on Cloud Environments
Icon
2. Cloud Deployments – Using Hadoop YARN on Cloud Environments
Icon
Introduction
Icon
Running Hadoop MapReduce v2 computations using Amazon Elastic MapReduce
Icon
Saving money using Amazon EC2 Spot Instances to execute EMR job flows
Icon
Executing a Pig script using EMR
Icon
Executing a Hive script using EMR
Icon
Creating an Amazon EMR job flow using the AWS Command Line Interface
Icon
Deploying an Apache HBase cluster on Amazon EC2 using EMR
Icon
Using EMR bootstrap actions to configure VMs for the Amazon EMR jobs
Icon
Using Apache Whirr to deploy an Apache Hadoop cluster in a cloud environment
3
3. Hadoop Essentials – Configurations, Unit Tests, and Other APIs
Icon
3. Hadoop Essentials – Configurations, Unit Tests, and Other APIs
Icon
Introduction
Icon
Optimizing Hadoop YARN and MapReduce configurations for cluster deployments
Icon
Shared user Hadoop clusters – using Fair and Capacity schedulers
Icon
Setting classpath precedence to user-provided JARs
Icon
Speculative execution of straggling tasks
Icon
Unit testing Hadoop MapReduce applications using MRUnit
Icon
Integration testing Hadoop MapReduce applications using MiniYarnCluster
Icon
Adding a new DataNode
Icon
Decommissioning DataNodes
Icon
Using multiple disks/volumes and limiting HDFS disk usage
Icon
Setting the HDFS block size
Icon
Setting the file replication factor
Icon
Using the HDFS Java API
4
4. Developing Complex Hadoop MapReduce Applications
Icon
4. Developing Complex Hadoop MapReduce Applications
Icon
Introduction
Icon
Choosing appropriate Hadoop data types
Icon
Implementing a custom Hadoop Writable data type
Icon
Implementing a custom Hadoop key type
Icon
Emitting data of different value types from a Mapper
Icon
Choosing a suitable Hadoop InputFormat for your input data format
Icon
Adding support for new input data formats – implementing a custom InputFormat
Icon
Formatting the results of MapReduce computations – using Hadoop OutputFormats
Icon
Writing multiple outputs from a MapReduce computation
Icon
Hadoop intermediate data partitioning
Icon
Secondary sorting – sorting Reduce input values
Icon
Broadcasting and distributing shared resources to tasks in a MapReduce job – Hadoop DistributedCache
Icon
Using Hadoop with legacy applications – Hadoop streaming
Icon
Adding dependencies between MapReduce jobs
Icon
Hadoop counters to report custom metrics
5
5. Analytics
Icon
5. Analytics
Icon
Introduction
Icon
Simple analytics using MapReduce
Icon
Performing GROUP BY using MapReduce
Icon
Calculating frequency distributions and sorting using MapReduce
Icon
Plotting the Hadoop MapReduce results using gnuplot
Icon
Calculating histograms using MapReduce
Icon
Calculating Scatter plots using MapReduce
Icon
Parsing a complex dataset with Hadoop
Icon
Joining two datasets using MapReduce
6
6. Hadoop Ecosystem – Apache Hive
Icon
6. Hadoop Ecosystem – Apache Hive
Icon
Introduction
Icon
Getting started with Apache Hive
Icon
Creating databases and tables using Hive CLI
Icon
Simple SQL-style data querying using Apache Hive
Icon
Creating and populating Hive tables and views using Hive query results
Icon
Utilizing different storage formats in Hive - storing table data using ORC files
Icon
Using Hive built-in functions
Icon
Hive batch mode - using a query file
Icon
Performing a join with Hive
Icon
Creating partitioned Hive tables
Icon
Writing Hive User-defined Functions (UDF)
Icon
HCatalog – performing Java MapReduce computations on data mapped to Hive tables
Icon
HCatalog – writing data to Hive tables from Java MapReduce computations
7
7. Hadoop Ecosystem II – Pig, HBase, Mahout, and Sqoop
Icon
7. Hadoop Ecosystem II – Pig, HBase, Mahout, and Sqoop
Icon
Introduction
Icon
Getting started with Apache Pig
Icon
Joining two datasets using Pig
Icon
Accessing a Hive table data in Pig using HCatalog
Icon
Getting started with Apache HBase
Icon
Data random access using Java client APIs
Icon
Running MapReduce jobs on HBase
Icon
Using Hive to insert data into HBase tables
Icon
Getting started with Apache Mahout
Icon
Running K-means with Mahout
Icon
Importing data to HDFS from a relational database using Apache Sqoop
Icon
Exporting data from HDFS to a relational database using Apache Sqoop
8
8. Searching and Indexing
Icon
8. Searching and Indexing
Icon
Introduction
Icon
Generating an inverted index using Hadoop MapReduce
Icon
Intradomain web crawling using Apache Nutch
Icon
Indexing and searching web documents using Apache Solr
Icon
Configuring Apache HBase as the backend data store for Apache Nutch
Icon
Whole web crawling with Apache Nutch using a Hadoop/HBase cluster
Icon
Elasticsearch for indexing and searching
Icon
Generating the in-links graph for crawled web pages
9
9. Classifications, Recommendations, and Finding Relationships
Icon
9. Classifications, Recommendations, and Finding Relationships
Icon
Introduction
Icon
Performing content-based recommendations
Icon
Classification using the naïve Bayes classifier
Icon
Assigning advertisements to keywords using the Adwords balance algorithm
10
10. Mass Text Data Processing
Icon
10. Mass Text Data Processing
Icon
Introduction
Icon
Data preprocessing using Hadoop streaming and Python
Icon
De-duplicating data using Hadoop streaming
Icon
Loading large datasets to an Apache HBase data store – importtsv and bulkload
Icon
Creating TF and TF-IDF vectors for the text data
Icon
Clustering text data using Apache Mahout
Icon
Topic discovery using Latent Dirichlet Allocation (LDA)
Icon
Document classification using Mahout Naive Bayes Classifier
11
Index
Icon
Index

Benchmarking HDFS using DFSIO

Hadoop contains several benchmarks that you can use to verify whether your HDFS cluster is set up properly and performs as expected. DFSIO is a benchmark test that comes with Hadoop, which can be used to analyze the I/O performance of an HDFS cluster. This recipe shows how to use DFSIO to benchmark the read/write performance of an HDFS cluster.

Getting ready

You must set up and deploy HDFS and Hadoop v2 YARN MapReduce prior to running these benchmarks. Locate the hadoop-mapreduce-client-jobclient-*-tests.jar file in your Hadoop installation.

How to do it...

The following steps will show you how to run the write and read DFSIO performance benchmarks:

  1. Execute the following command to run the HDFS write performance benchmark. The –nrFiles parameter specifies the number of files to be written by the benchmark. Use a number high enough to saturate the task slots in your cluster. The -fileSize parameter specifies the file size of each file in MB. Change the location of the hadoop-mapreduce-client-jobclient-*-tests.jar file in the following commands according to your Hadoop installation.
    $ hadoop jar \
$HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-*-tests.jar \
TestDFSIO -write -nrFiles 32 –fileSize 1000
  2. The write benchmark writes the results to the console as well as appending to a file named TestDFSIO_results.log. You can provide your own result filename using the –resFile parameter.
  3. The following step will show you how to run the HDFS read performance benchmark. The read performance benchmark uses the files written by the write benchmark in step 1. Hence, the write benchmark should be executed before running the read benchmark and the files written by the write benchmark should exist in the HDFS for the read benchmark to work properly. The benchmark writes the results to the console and appends the results to a logfile similarly to the write benchmark.
    $hadoop jar \
$HADOOP_HOME/share/Hadoop/mapreduce/hadoop-mapreduce-client-jobclient-*-tests.jar \
TestDFSIO -read \
-nrFiles 32 –fileSize 1000
  4. The files generated by the preceding benchmarks can be cleaned up using the following command:
    $hadoop jar \
$HADOOP_HOME/share/Hadoop/mapreduce/hadoop-mapreduce-client-jobclient-*-tests.jar \
TestDFSIO -clean

How it works...

DFSIO executes a MapReduce job where the Map tasks write and read the files in parallel, while the Reduce tasks are used to collect and summarize the performance numbers. You can compare the throughput and IO rate results of this benchmark with the total number of disks and their raw speeds to verify whether you are getting the expected performance from your cluster. Please note the replication factor when verifying the write performance results. High standard deviation in these tests may hint at one or more underperforming nodes due to some reason.

There's more...

Running these tests together with monitoring systems can help you identify the bottlenecks of your Hadoop cluster much easily.

Visually different images
CONTINUE READING
83
Tech Concepts
36
Programming languages
73
Tech Tools
Icon Unlimited access to the largest independent learning library in tech of over 8,000 expert-authored tech books and videos.
Icon Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.
Icon 50+ new titles added per month and exclusive early access to books as they are being written.
Hadoop MapReduce v2 Cookbook - Second Edition: RAW
End of Section 1
Next Section
notes
bookmark Notes and Bookmarks search Search in title playlist Add to playlist font-size Font size

Change the font size

margin-width Margin width

Change margin width

day-mode Day/Sepia/Night Modes

Change background colour

Close icon Search
Country selected
Close icon Your notes and bookmarks

Confirmation

Modal Close icon
claim successful
Order Page Read Now

Buy this book with your credits?

Modal Close icon
Are you sure you want to buy this book with one of your credits?
Close
YES, BUY

Submit Your Feedback

Modal Close icon
Modal Close icon
Modal Close icon