Book Image

Learning Big Data with Amazon Elastic MapReduce

By : Amarkant Singh, Vijay Rayapati
Book Image

Learning Big Data with Amazon Elastic MapReduce

By: Amarkant Singh, Vijay Rayapati

Overview of this book

<p>Amazon Elastic MapReduce is a web service used to process and store vast amount of data, and it is one of the largest Hadoop operators in the world. With the increase in the amount of data generated and collected by many businesses and the arrival of cost-effective cloud-based solutions for distributed computing, the feasibility to crunch large amounts of data to get deep insights within a short span of time has increased greatly.</p> <p>This book will get you started with AWS so that you can quickly create your own account and explore the services provided, many of which you might be delighted to use. This book covers the architectural details of the MapReduce framework, Apache Hadoop, various job models on EMR, how to manage clusters on EMR, and the command-line tools available with EMR. Each chapter builds on the knowledge of the previous one, leading to the final chapter where you will learn about solving a real-world use case using Apache Hadoop and EMR. This book will, therefore, get you up and running with major Big Data technologies quickly and efficiently.</p>
Table of Contents (18 chapters)
Learning Big Data with Amazon Elastic MapReduce
Credits
About the Authors
Acknowledgments
About the Reviewers
www.PacktPub.com
Preface
Index

EMR cluster monitoring and troubleshooting


We can use one of the tools (web console, CLI, SDK, or API) to get EMR cluster details in AWS. The web console displays all of the clusters you've launched in the past two weeks (both active and terminated).

We have seen in the previous chapter that if you click on a cluster name, then the web console displays a Details pane with information about that cluster. As we will see in our next chapter, we can also find the details about a cluster from the CLI using the --describe argument along with a Job Flow ID.

EMR cluster logging

Amazon EMR and Hadoop both generate logfiles as the cluster begins execution. You can access these logfiles from several different tools, depending on the configuration specified when we launch the cluster.

Every cluster publishes log files to the /mnt/var/log/ directory on the master node. These logfiles are only available while the cluster is running.

When you launch the cluster with an Amazon S3 log path, the cluster copies...