In a short span of time, YARN has attained a great deal of momentum and acceptance in the big data world.
YARN essentials is about YARN—the modern operating system for Hadoop. This book contains all that you need to know about YARN, right from its inception to the present and future.
In the first part of the book, you will be introduced to the motivation behind the development of YARN and learn about its core architecture, installation, and administration. This part also talks about the architectural differences that YARN brings to Hadoop 2 with respect to Hadoop 1 and why this redesign was needed.
In the second part, you will learn how to write a YARN application, how to submit an application to YARN, and how to monitor the application. Next, you will learn about the various emerging open source frameworks that are developed to run on top of YARN. You will learn to develop and deploy some use case examples using Apache Samza and Storm YARN.
Finally, we will talk about the failures in YARN, some alternative solutions available on the market, and the future and support for YARN in the big data world.
Chapter 1, Need for YARN, discusses the motivation behind the development of YARN. This chapter discusses what YARN is and why it is needed.
Chapter 2, YARN Architecture, is a deep dive into YARN's architecture. All the major components and their inner workings are explained in this chapter.
Chapter 3, YARN Installation, describes the steps required to set up a single-node and fully-distributed YARN cluster. It also talks about the important configurations/properties that you should be aware of while installing the YARN cluster.
Chapter 4, YARN and Hadoop Ecosystems, talks about Hadoop with respect to YARN. It gives a short introduction to the Hadoop 1.x version, the architectural differences between Hadoop 1.x and Hadoop 2.x, and where exactly YARN fits into Hadoop 2.x.
Chapter 5, YARN Administration, covers information on the administration of YARN clusters. It explains the administrative tools that are available in YARN, what they mean, and how to use them. This chapter covers various topics from YARN container allocation and configuration to various scheduling policies/configurations and in-built support for multitenancy.
Chapter 6, Developing and Running a Simple YARN Application, focuses on some real applications with YARN, with some hands-on examples. It explains how to write a YARN application, how to submit an application to YARN, and finally, how to monitor the application.
Chapter 7, YARN Frameworks, discusses the various emerging open source frameworks that are developed to run on top of YARN. The chapter then talks in detail about Apache Samza and Storm on YARN, where we will develop and run some sample applications using these frameworks.
Chapter 8, Failures in YARN, discusses the fault-tolerance aspect of YARN. This chapter focuses on various failures that can occur in the YARN framework, their causes, and how YARN gracefully handles those failures.
Chapter 9, YARN – Alternative Solutions, discusses other alternative solutions that are available on the market today. These systems, like YARN, share common inspiration/requirements and the high-level goal of improving scalability, latency, fault-tolerance, and programming model flexibility. This chapter highlights the key differences in the way these alternative solutions address the same features provided by YARN.
Chapter 10, YARN Future and Support, talks about YARN's journey and its present and future in the world of distributed computing.
You will need a single Linux-based machine with JDK 1.6 or later installed. Any recent version of the Apache Hadoop 2 distribution will be sufficient to set up a YARN cluster and run some examples on top of YARN.
The code in this book has been tested on CentOS 6.4 but will run on other variants of Linux.
This book is for the big data enthusiasts who want to gain in-depth knowledge of YARN and know what really makes YARN the modern operating system for Hadoop. You will develop a good understanding of the architectural differences that YARN brings to Hadoop 2 with respect to Hadoop 1.
You will develop in-depth knowledge about the architecture and inner workings of the YARN framework.
After finishing this book, you will be able to install, administrate, and develop YARN applications. This book tells you anything you need to know about YARN, right from its inception to its present and future in the big data industry.
In this book, you will find a number of text styles that distinguish between different kinds of information. Here are some examples of these styles and an explanation of their meaning.
Code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles are shown as follows: " The URL for NameNode is http://<namenode_host>:<port>/ and the default HTTP port is 50070."
A block of code is set as follows:
<property>
<name>io.file.buffer.size</name>
<value>4096</value>
<description>read and write buffer size of files</description>
</property>Any command-line input or output is written as follows:
${path_to_your_input_dir} ${path_to_your_output_dir_old}
New terms and important words are shown in bold. Words that you see on the screen, for example, in menus or dialog boxes, appear in the text like this: "Under the Tools section, you can find the YARN configuration file details, scheduling information, container configurations, local logs of the jobs, and a lot of other information on the cluster."
Feedback from our readers is always welcome. Let us know what you think about this book—what you liked or disliked. Reader feedback is important for us as it helps us develop titles that you will really get the most out of.
To send us general feedback, simply e-mail <[email protected]>, and mention the book's title in the subject of your message.
If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, see our author guide at www.packtpub.com/authors.
Now that you are the proud owner of a Packt book, we have a number of things to help you to get the most from your purchase.
You can download the example code files from your account at http://www.packtpub.com for all the Packt Publishing books you have purchased. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you.
Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you find a mistake in one of our books—maybe a mistake in the text or the code—we would be grateful if you could report this to us. By doing so, you can save other readers from frustration and help us improve subsequent versions of this book. If you find any errata, please report them by visiting http://www.packtpub.com/submit-errata, selecting your book, clicking on the Errata Submission Form link, and entering the details of your errata. Once your errata are verified, your submission will be accepted and the errata will be uploaded to our website or added to any list of existing errata under the Errata section of that title.
To view the previously submitted errata, go to https://www.packtpub.com/books/content/support and enter the name of the book in the search field. The required information will appear under the Errata section.
Piracy of copyrighted material on the Internet is an ongoing problem across all media. At Packt, we take the protection of our copyright and licenses very seriously. If you come across any illegal copies of our works in any form on the Internet, please provide us with the location address or website name immediately so that we can pursue a remedy.
Please contact us at <[email protected]> with a link to the suspected pirated material.
We appreciate your help in protecting our authors and our ability to bring you valuable content.
If you have a problem with any aspect of this book, you can contact us at <[email protected]>, and we will do our best to address the problem.