Today, many organizations are facing the Big Data problem. Managing and processing Big Data can incur a lot of challenges for traditional data processing platforms such as relational database systems. Hadoop was designed to be a distributed and scalable system for dealing with Big Data problems. A Hadoop-based Big Data platform uses Hadoop as the data storage and processing engine. It deals with the problem by transforming the Big Data input into expected output.
Hadoop Operations and Cluster Management Cookbook provides examples and step-by-step recipes for you to administrate a Hadoop cluster. It covers a wide range of topics for designing, configuring, managing, and monitoring a Hadoop cluster. The goal of this book is to help you manage a Hadoop cluster more efficiently and in a more systematic way.
In the first three chapters, you will learn practical recipes to configure a fully distributed Hadoop cluster. The subsequent management, hardening, and performance tuning chapters will cover the core topics of this book. In these chapters, you will learn practical commands and best practices to manage a Hadoop cluster. The last important topic of the book is the monitoring of a Hadoop cluster. And, we will end this book by introducing steps to build a Hadoop cluster using the AWS cloud.
Chapter 1, Big Data and Hadoop, introduces steps to define a Big Data problem and outlines steps to build a Hadoop-based Big Data platform.
Chapter 2, Preparing for Hadoop Installation, describes the preparation of a Hadoop cluster configuration. Topics include choosing the proper cluster hardware, configuring the network, and installing the Linux operating system.
Chapter 3, Configuring a Hadoop Cluster, introduces recipes to configure a Hadoop cluster in pseudo-distributed mode as well as in fully distributed mode. We will also describe steps to verify and troubleshoot a Hadoop cluster configuration.
Chapter 4, Managing a Hadoop Cluster, shows you how to manage a Hadoop cluster. We will learn cluster maintenance tasks and practical steps to do the management. For example, we will introduce the management of an HDFS filesystem, management of MapReduce jobs, queues and quota, and so on.
Chapter 5, Hardening a Hadoop Cluster, introduces recipes to secure a Hadoop cluster. We will show you how to configure ACL for authorization and Kerberos for authentication, configure NameNode HA, recover from a failed NameNode, and so on.
Chapter 6, Monitoring a Hadoop Cluster, explains how to monitor a Hadoop cluster with various tools, such as Ganglia and Nagios.
Chapter 7, Tuning a Hadoop Cluster for Best Performance, introduces best practices to tune the performance of a Hadoop cluster. We will tune the memory profile, the MapReduce scheduling strategy, and so on to achieve best performance for a Hadoop cluster.
Chapter 8, Building a Hadoop Cluster with Amazon EC2 and S3, shows you how to configure a Hadoop cluster in the Amazon cloud. We will explain steps to register, connect, and start VM instances on EC2. We will also show you how to configure a customized AMI for a Hadoop cluster on EC2.
This book is written to be as self-contained as possible. Each chapter and recipe has its specific prerequisites introduced before the topic.
In general, in this book, we will use the following software packages:
CentOS 6.3
Oracle JDK (Java Development Kit) SE 7
Hadoop 1.1.2
HBase 0.94.5
Hive 0.9.0
Pig 0.10.1
ZooKeeper 3.4.5
Mahout 0.7
This book is for Hadoop administrators and Big Data architects. It can be a helpful book for Hadoop programmers.
You are not required to have solid knowledge about Hadoop to read this book, but you are required to know basic Linux commands and have a general understanding of distributed computing concepts.
In this book, you will find a number of styles of text that distinguish between different kinds of information. Here are some examples of these styles, and an explanation of their meaning.
Code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles are shown as follows: "Open the file $HADOOP_HOME/conf/mapred-site.xml with your favorite text editor."
A block of code is set as follows:
<property> <name>fs.default.name</name> <value>hdfs://master:54310</value> </property>
When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold:
<property>
<name>fs.default.name</name>
<value>hdfs://master:54310</value>
</property>Any command-line input or output is written as follows:
hadoop namenode -format
New terms and important words are shown in bold. Words that you see on the screen, in menus or dialog boxes for example, appear in the text like this: "By clicking on the link Analyze This Job, we will go to a web page."
Feedback from our readers is always welcome. Let us know what you think about this book—what you liked or may have disliked. Reader feedback is important for us to develop titles that you really get the most out of.
To send us general feedback, simply send an e-mail to <[email protected]>, and mention the book title via the subject of your message.
If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, see our author guide on www.packtpub.com/authors.
Now that you are the proud owner of a Packt book, we have a number of things to help you to get the most from your purchase.
Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you find a mistake in one of our books—maybe a mistake in the text or the code—we would be grateful if you would report this to us. By doing so, you can save other readers from frustration and help us improve subsequent versions of this book. If you find any errata, please report them by visiting http://www.packtpub.com/submit-errata, selecting your book, clicking on the errata submission form link, and entering the details of your errata. Once your errata are verified, your submission will be accepted and the errata will be uploaded on our website, or added to any list of existing errata, under the Errata section of that title. Any existing errata can be viewed by selecting your title from http://www.packtpub.com/support.
Piracy of copyright material on the Internet is an ongoing problem across all media. At Packt, we take the protection of our copyright and licenses very seriously. If you come across any illegal copies of our works, in any form, on the Internet, please provide us with the location address or website name immediately so that we can pursue a remedy.
Please contact us at <[email protected]> with a link to the suspected pirated material.
We appreciate your help in protecting our authors, and our ability to bring you valuable content.
You can contact us at <[email protected]> if you are having a problem with any aspect of the book, and we will do our best to address it.