With the increasing popularity of Big Data in enterprise, every day more and more workloads are being shifted to Hadoop.
To run those regular processing jobs on Hadoop, we need a scheduler that can act as cron for all data pipelines. Oozie plays this role in the Big Data world.
This book introduces you to the world of Oozie using a step-by-step case study-based approach.
Chapter 1, Setting up Oozie, covers how to install and configure Oozie in Hadoop cluster. We will also learn how to install Oozie from the source code.
Chapter 2, My First Oozie Job, covers running a "Hello World" equivalent first Oozie job. It also introduces the concept of Workflow, Coordinator, and Bundles.
Chapter 3, Oozie Fundamentals, introduces the fundamental concepts of control nodes, expression language, web console, and running Oozie jobs from Hue.
Chapter 4, Running MapReduce Jobs, teaches how to run MapReduce jobs from Oozie and explores the concepts of Coordinators, Datasets, and cron-based frequency schedules.
Chapter 5, Running Pig Jobs, teaches how to run Pig jobs from Oozie. We will also cover the concept of parameterization of Datasets and Coordinator controls.
Chapter 6, Running Hive Jobs, introduces how to run Hive jobs and discusses the concepts of parameterization of Coordinator actions.
Chapter 7, Running Sqoop Jobs, shows how to run Sqoop jobs from Oozie and introduces the concept of HCatalog Datasets and EL functions.
Chapter 8, Running Spark Jobs, shows how to run Spark jobs. It also introduces the concept of Bundles and how they are used to group a set of Coordinator jobs.
Chapter 9, Running Oozie in Production, covers how to package the code for production deployments and how to rerun the jobs that have failed.
To follow the tutorial and code examples in this book, you need to have access to Hadoop cluster or you can configure a single node virtual machine-based cluster. You should have a good laptop/desktop, preferably with a Linux operating system or Windows with VirtualBox installed.
This book is for anyone who is familiar with basics of Hadoop and Hive, and now wants to automate the data and machine learning pipelines using Apache Oozie.
In this book, you will find a number of text styles that distinguish between different kinds of information. Here are some examples of these styles and an explanation of their meaning.
Code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles are shown as follows: "Now, edit the torrc file placed at the /etc/tor/ directory."
Most of the code in the book is XML. A block of code is set as follows:
<workflow-app name="My_first_Workflow" xmlns="uri:oozie:workflow:0.5">
<start to="fs-2178"/>
<kill name="Kill">
<message>Action failed </message>
</kill>
<action name="fs-2178">
<fs>
<delete path='${nameNode}/user/hue'/>
</fs>
<ok to="End"/>
<error to="Kill"/>
</action>
<end name="End"/>
</workflow-app>Any command-line input or output is written as follows:
# $ hadoop fs -ls /user/hue/learn_oozie
New terms and important words are shown in bold. Words that you see on the screen, for example, in menus or dialog boxes, appear in the text like this: "Go to Settings | Networking | Port Forwarding , Click on Add new port forwarding."
Feedback from our readers is always welcome. Let us know what you think about this book—what you liked or disliked. Reader feedback is important for us as it helps us develop titles that you will really get the most out of.
To send us general feedback, simply e-mail <[email protected]>, and mention the book's title in the subject of your message.
If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, see our author guide at www.packtpub.com/authors.
Now that you are the proud owner of a Packt book, we have a number of things to help you to get the most from your purchase.
You can download the example code files from your account at http://www.packtpub.com for all the Packt Publishing books you have purchased. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you.
Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you find a mistake in one of our books—maybe a mistake in the text or the code—we would be grateful if you could report this to us. By doing so, you can save other readers from frustration and help us improve subsequent versions of this book. If you find any errata, please report them by visiting http://www.packtpub.com/submit-errata, selecting your book, clicking on the Errata Submission Form link, and entering the details of your errata. Once your errata are verified, your submission will be accepted and the errata will be uploaded to our website or added to any list of existing errata under the Errata section of that title.
To view the previously submitted errata, go to https://www.packtpub.com/books/content/support and enter the name of the book in the search field. The required information will appear under the Errata section.
Piracy of copyrighted material on the Internet is an ongoing problem across all media. At Packt, we take the protection of our copyright and licenses very seriously. If you come across any illegal copies of our works in any form on the Internet, please provide us with the location address or website name immediately so that we can pursue a remedy.
Please contact us at <[email protected]> with a link to the suspected pirated material.
We appreciate your help in protecting our authors and our ability to bring you valuable content.
If you have a problem with any aspect of this book, you can contact us at <[email protected]>, and we will do our best to address the problem.