Book Image

Learning Big Data with Amazon Elastic MapReduce

By : Amarkant Singh, Vijay Rayapati
Book Image

Learning Big Data with Amazon Elastic MapReduce

By: Amarkant Singh, Vijay Rayapati

Overview of this book

<p>Amazon Elastic MapReduce is a web service used to process and store vast amount of data, and it is one of the largest Hadoop operators in the world. With the increase in the amount of data generated and collected by many businesses and the arrival of cost-effective cloud-based solutions for distributed computing, the feasibility to crunch large amounts of data to get deep insights within a short span of time has increased greatly.</p> <p>This book will get you started with AWS so that you can quickly create your own account and explore the services provided, many of which you might be delighted to use. This book covers the architectural details of the MapReduce framework, Apache Hadoop, various job models on EMR, how to manage clusters on EMR, and the command-line tools available with EMR. Each chapter builds on the knowledge of the previous one, leading to the final chapter where you will learn about solving a real-world use case using Apache Hadoop and EMR. This book will, therefore, get you up and running with major Big Data technologies quickly and efficiently.</p>
Table of Contents (18 chapters)
Learning Big Data with Amazon Elastic MapReduce
Credits
About the Authors
Acknowledgments
About the Reviewers
www.PacktPub.com
Preface
Index

Emitting results to multiple outputs


Hadoop provides a utility class org.apache.hadoop.mapreduce.lib.output.MultipleOutputs that simplifies writing output data to multiple files and locations.

You might use MultipleOutputs mainly for the following two use cases:

  • To emit additional outputs other than the job default output

  • To emit data to different files and/or directories provided by a user

Using MultipleOutputs

Each additional output or named output might be configured with its own output format, key class, and value class. You can define multiple named outputs in your Driver class and then use them in your Reducer class to emit additional output. Each Reducer class creates a separate copy of these named outputs, the same way it does with default job output.

Usage in the Driver class

You can use the following static method of the MutilpleOutputs class to define a named output, which will be in addition to the job default output:

addNamedOutput(Job job, String namedOutput, Class<? extends OutputFormat...