Book Image

Learning Big Data with Amazon Elastic MapReduce

By : Amarkant Singh, Vijay Rayapati
Book Image

Learning Big Data with Amazon Elastic MapReduce

By: Amarkant Singh, Vijay Rayapati

Overview of this book

<p>Amazon Elastic MapReduce is a web service used to process and store vast amount of data, and it is one of the largest Hadoop operators in the world. With the increase in the amount of data generated and collected by many businesses and the arrival of cost-effective cloud-based solutions for distributed computing, the feasibility to crunch large amounts of data to get deep insights within a short span of time has increased greatly.</p> <p>This book will get you started with AWS so that you can quickly create your own account and explore the services provided, many of which you might be delighted to use. This book covers the architectural details of the MapReduce framework, Apache Hadoop, various job models on EMR, how to manage clusters on EMR, and the command-line tools available with EMR. Each chapter builds on the knowledge of the previous one, leading to the final chapter where you will learn about solving a real-world use case using Apache Hadoop and EMR. This book will, therefore, get you up and running with major Big Data technologies quickly and efficiently.</p>
Table of Contents (18 chapters)
Learning Big Data with Amazon Elastic MapReduce
Credits
About the Authors
Acknowledgments
About the Reviewers
www.PacktPub.com
Preface
Index

The reduce function


Before the reduce function comes into the picture, the MapReduce framework groups all the map output as per the key and lines up the input for the reducer. It is again formed as a key-value pair wherein the key is the same as the output of the map function, but the value is now the list of values of all those map outputs having the same key. All the preceding map outputs now get grouped as follows:

<Country1, [1,1,1,1]>
<Country2, [1,1]>
<Country3, [1,1,1]>

Each of these above aggregated key-value pairs will be fed to the reduce function. A reduce function expects a key and a list of values as input. Now, we have a super simple task of addition in the reduce function. This function just needs to add up the number of elements in the list of values it receives and emit the output key-value pair in the form of <Country, Number-of-occurrences>. Even reduce functions are stateless as they are isolated from the execution of other reduce functions. So,...