Many organizations are implementing Hadoop in production environments, storing critical data on it, and making sure everything is in place and running as desired as it is crucial for the business. If something breaks down, how quickly you can detect it and remediate it is very important. In order to have early detection of any failures, there is a need to have monitoring in place and capture events that let you peep into the internal workings of a Hadoop cluster. The goal of this book is to enable monitoring and capture events to make sure that the Hadoop clusters are up and running to the optimal capacity.
Chapter 1, Introduction to Monitoring, discusses the need for monitoring and the tools available in the market for that. This chapter also provides details about installing Nagios and Ganglia, which are the tools to monitor and capture metrics for a Hadoop cluster.
Chapter 2, Hadoop Daemons and Services, discusses the Hadoop services and daemons and how they communicate. Before implementing monitoring, one must understand how Hadoop components talk to each other and what ports the services run on.
Chapter 3, Hadoop Logging, discusses how system logging works and how that extends to logging in Hadoop clusters. This chapter also covers the logging details for various Hadoop daemons.
Chapter 4, HDFS Checks, explores the HDFS checks, which can be implemented for Hadoop File System and its components, such as NameNode, DataNode, and so on.
Chapter 5, MapReduce Checks, discusses configuring checks for MapReduce components, such as JobTracker, TaskTracker, ResourceManager, and other YARN components.
Chapter 6, Hadoop Metrics and Visualization Using Ganglia, provides a step-by-step guide to configuring a Hadoop metrics collection and its visualization using Ganglia.
Chapter 7, Hive, HBase, and Monitoring Best Practices, provides an introduction to metrics collection and monitoring for the Hive and HBase components of the Hadoop framework. It also talks about the best practices for monitoring on a large scale and how to keep the utilization of the monitoring servers optimized.
To practice the examples provided in this book, you will need a working Hadoop cluster. It is recommended that you use Cent OS 6.0 at the minimum and Apache Hadoop 1.2.1 and Hadoop 2.6.0 for the Hadoop version 1 and Hadoop version 2 examples, respectively.
Monitoring Hadoop is ideal for Hadoop administrators who need to monitor their Hadoop clusters and make sure they are running optimally. This book acts as a reference to set up Hadoop monitoring and visualization using Ganglia.
In this book, you will find a number of text styles that distinguish between different kinds of information. Here are some examples of these styles and an explanation of their meaning.
Code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles are shown as follows: "This is the port for ResourceManager scheduler; the default is 8030."
A block of code is set as follows:
log4j.appender.DRFAAUDIT=org.apache.log4j.DailyRollingFileAppender log4j.appender.DRFAAUDIT.File=/var/log/audit.log log4j.appender.DRFAAUDIT.DatePattern=.yyyy-MM-dd log4j.appender.DRFAAUDIT.layout=org.apache.log4j.PatternLayout
When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold:
log4j.appender.DRFAAUDIT=org.apache.log4j.DailyRollingFileAppender log4j.appender.DRFAAUDIT.File=/var/log/audit.log log4j.appender.DRFAAUDIT.DatePattern=.yyyy-MM-dd log4j.appender.DRFAAUDIT.layout=org.apache.log4j.PatternLayout
Any command-line input or output is written as follows:
$ sudo /usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg
New terms and important words are shown in bold. Words that you see on the screen, for example, in menus or dialog boxes, appear in the text like this: "If you see a message, such as Return code of 127 is out of bounds – plugin may be missing on the right panel, then this means that your configuration is correct as of now."
Feedback from our readers is always welcome. Let us know what you think about this book—what you liked or disliked. Reader feedback is important for us as it helps us develop titles that you will really get the most out of.
To send us general feedback, simply e-mail <[email protected]>, and mention the book's title in the subject of your message.
If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, see our author guide at www.packtpub.com/authors.
Now that you are the proud owner of a Packt book, we have a number of things to help you to get the most from your purchase.
Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you find a mistake in one of our books—maybe a mistake in the text or the code—we would be grateful if you could report this to us. By doing so, you can save other readers from frustration and help us improve subsequent versions of this book. If you find any errata, please report them by visiting http://www.packtpub.com/submit-errata, selecting your book, clicking on the Errata Submission Form link, and entering the details of your errata. Once your errata are verified, your submission will be accepted and the errata will be uploaded to our website or added to any list of existing errata under the Errata section of that title.
To view the previously submitted errata, go to https://www.packtpub.com/books/content/support and enter the name of the book in the search field. The required information will appear under the Errata section.
Piracy of copyrighted material on the Internet is an ongoing problem across all media. At Packt, we take the protection of our copyright and licenses very seriously. If you come across any illegal copies of our works in any form on the Internet, please provide us with the location address or website name immediately so that we can pursue a remedy.
Please contact us at <[email protected]> with a link to the suspected pirated material.
We appreciate your help in protecting our authors and our ability to bring you valuable content.
If you have a problem with any aspect of this book, you can contact us at <[email protected]>, and we will do our best to address the problem.